Method and apparatus for recognizing and performing handwritten calculations

ABSTRACT

A pen-based calculator recognizes handwritten input. The calculator comprises a display simulating a sheet of paper, and a stylus simulating a pen. The user writes a calculation on the calculator as if it were a piece of scratch paper. The calculator uses handwriting recognition to identify the various elements of the calculation, performs the calculation, and then displays the result at an appropriate location.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to calculators, and more specifically,to a calculator capable of recognizing handwritten input.

2. Description of Related Art

At times, nearly everyone still uses scratch paper to perform manualcalculations. These are skills that are taught at an early age.Typically, a person writes the numbers on the paper in a horizontalarrangement with an equal sign:

111+222=333

Alternatively, the person writes numbers on paper in a verticalarrangement with decimals aligned and a line underneath:

    ______________________________________                                          111                                                                         + 222                                                                           333                                                                         ______________________________________                                    

The person does the calculation manually using techniques learned earlyin life, and then writes the result (which is sometimes wrong) below theline or to the right of the equal sign. If more calculations arerequired using the result, then the person places them near the resultto avoid writing twice. Some information may be left off the paper. Theperson can copy numbers from one part of the paper to another. The paperremembers everything done, and displays the whole calculation. Withsufficient training or tools, anyone can do any mathematical operation.

Because of the drudgery involved and because of the real possibility oferrors, many calculations are performed with calculators. Calculatorsare typically mechanical or electric devices. Electronic calculators arefast and accurate, and the advent of integrated circuits have providedmore powerful calculators capable of sophisticated functions, includingprogrammable calculators. Today, calculators can be specialized forparticular applications including basic engineering, scientific orstatistical applications.

Unfortunately, as calculators become more powerful and capable, theybecome harder to use. Often, function keys on the calculator mayrepresent two, three or more functions. A person has to read andunderstand a thick reference manual in order to use the calculator. Nowonder people may believe you need a Ph.D just to operate thecalculator.

The present invention takes the concept of a calculator in a full circleback to its beginnings, i.e., the use of scratch paper. The presentinvention simulates a "smart pad" of scratch paper capable of performingcalculations that a person handwrites in a natural notation. The presentinvention recognizes all types of calculations and mathematicalfunctions, thereby providing the same capabilities as the most powerfulcalculators.

SUMMARY OF THE INVENTION

To overcome the limitations in the prior art described above, and toovercome other limitations that will become apparent upon reading andunderstanding the present specification, the present invention disclosesa pen-based calculator that recognizes handwritten input. The calculatorcomprises a display simulating a sheet of paper, and a stylus simulatinga pen. The user writes a calculation on the calculator as if it were apiece of scratch paper. The calculator uses handwriting recognition toidentify the various elements of the calculation, performs thecalculation, and then displays the result at an appropriate location.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 illustrates the packaging for a SmartPad calculator according tothe present invention;

FIG. 2 illustrates the electronic components comprising the presentinvention;

FIG. 3 illustrates the scratch paper metaphor as presented to the useron the screen of the SmartPad calculator;

FIG. 4 illustrates a "training palette" as presented to the user on thescreen of the SmartPad calculator;

FIG. 5 illustrates counter functions as presented to the user on thescreen of the SmartPad calculator;

FIG. 6 illustrates conditional functions as presented to the user on thescreen of the SmartPad calculator;

FIG. 7 illustrates spreadsheet functions as presented to the user on thescreen of the SmartPad calculator;

FIG. 8 illustrates a copy operation as presented to the user on thescreen of the SmartPad calculator;

FIG. 9 illustrates total and subtotal calculations as presented to theuser on the screen of the SmartPad calculator;

FIG. 10 illustrates calculations linked indirectly through a copyoperation as presented to the user on the screen of the SmartPadcalculator;

FIGS. 11A and 11B illustrate inter-page links between calculations aspresented to the user on the screen of the SmartPad calculator;

FIGS. 12A and 12B illustrate a selection operation as presented to theuser on the screen of the SmartPad calculator;

FIGS. 13A, 13B, and 13C illustrate how corrections are made on thescreen of the SmartPad calculator by overwriting the number;

FIGS. 14A, 14B, 14C, 14D, and 14E illustrate how corrections are made onthe screen of the SmartPad calculator by inserting digits in the number;

FIGS. 15A, 15B, and 15C illustrate how corrections are made on thescreen of the SmartPad calculator by deleting digits from the numbers;

FIGS. 16A, 16B, and 16C illustrate how corrections are made on thescreen of the SmartPad calculator by interchanging digits in thenumbers;

FIGS. 17A, 17B, 17C, 17D, and 17E illustrate how corrections are made onthe screen of the SmartPad calculator by erasing digits from the numbersand replacing them with new digits;

FIGS. 18A, 18B, and 18C illustrate how corrections are made on thescreen of the SmartPad calculator by overwriting the Operator;

FIGS. 19A, 19B, 19C, 19D, 19E, and 19F illustrate how corrections aremade on the screen of the SmartPad calculator by inserting operators andnumbers;

FIGS. 20A, 20B, and 20C illustrate how corrections are made on thescreen of the SmartPad calculator by deleting operators and numbers fromthe calculations;

FIGS. 21A, 21B, 21C, 21D, and 21E illustrate how corrections are made onthe screen of the SmartPad calculator by replacing operators and numbersin the calculations;

FIGS. 22A, 22B, and 22C illustrate how calculations are stopped on thescreen of the SmartPad calculator by erasing the result and/orcalculating symbol;

FIGS. 23A, 23B, and 23C illustrate how corrections are made on thescreen of the SmartPad calculator by interchanging operators and numbersin the calculations;

FIGS. 24A, 24B, 24C, 24D, and 24E illustrate how corrections are made onthe screen of the SmartPad calculator by selecting a part thereof andthen moving it to a new position;

FIGS. 25A and 25B illustrate how links are established on the screen ofthe SmartPad calculator by position or by copying;

FIGS. 26A, 26B, 26C, 26D, and 26E illustrate how position links arebroken on the screen of the SmartPad calculator;

FIGS. 27A, 27B, and 27C illustrate how copy links are broken on thescreen of the SmartPad calculator by erasing or deleting the copy;

FIGS. 28A, 28B, 28C, 28D, and 28E illustrate how copy links are brokenon the screen of the SmartPad calculator by selecting and moving thecopy out of the second calculation;

FIGS. 29A, 29B, 29C, 29D, and 29E illustrate how position links are madeon the screen of the SmartPad calculator;

FIGS. 30A, 30B, and 30C illustrate how copy links are made on the screenof the SmartPad calculator;

FIGS. 31A, 31B, 31C, and 31D illustrate how label corrections are madeon the screen of the SmartPad calculator;

FIGS. 32A, 32B, 32C, 32D, 32E and 32F illustrate how object positionchanges are made on the screen of the SmartPad calculator;

FIGS. 33A, 33B, and 33C illustrate how to move numerous objects by justmoving one result on the screen of the SmartPad calculator;

FIGS. 34A, 34B, and 34C illustrate how the move of the original or copydoes not affect the position of the other object on the screen of theSmartPad calculator;

FIGS. 35A, 35B, 35C, 35D, 35E, and 35F illustrate how changes to thesize of objects are made on the screen of the SmartPad calculator;

FIGS. 36A, 36B, 36C, and 36D illustrate how changes are made to theorientation of the screen of the SmartPad calculator;

FIGS. 37A, 37B, 37C, and 37D illustrate how scrolling is performed onthe screen of the SmartPad calculator;

FIG. 38 illustrates a table of contents page for a pad as presented tothe user on the screen of the SmartPad calculator;

FIGS. 39A, 39B, 39C, and 39D illustrate how pages can be split on thescreen of the SmartPad calculator;

FIG. 40 is a flow chart describing the logic of the Program function inthe preferred embodiment of the SmartPad calculator;

FIG. 41 is a flow chart describing the logic of the Initialize functionin the SmartPad calculator;

FIGS. 42A and 42B together are a flow chart describing the logic of theUpdate-Page function in the SmartPad calculator;

FIG. 43 is a flow chart describing the logic of the Respond-to-Penfunction in the SmartPad calculator;

FIG. 44 is a flow chart describing the logic of the Handle-Pen-In-Titlefunction in the SmartPad calculator;

FIGS. 45A and 45B together are a flow chart describing the logic of theHandle-Pen-In-Palette function in the SmartPad calculator;

FIGS. 46A and 46B together are a flow chart describing the logic of theHandle-Pen-In-Menu function in the SmartPad calculator;

FIGS. 47A and 47B together are a flow chart describing the logic of theHandle-Pen-In-Page function in the SmartPad calculator;

FIG. 48 is a flow chart describing the logic of theHandle-Written-Symbol function in the SmartPad calculator;

FIGS. 49A and 49B together are a flow chart describing the logic of theDo-Action function in the SmartPad calculator;

FIGS. 50A, 50B, 50C, and 50D together are a flow chart describing thelogic of the Add-Object function in the SmartPad calculator;

FIGS. 51A and 51B together are a flow chart describing the logic of theGenerate-Number function in the SmartPad calculator;

FIG. 52 is a flow chart describing the logic of theGenerate-Horizontal-Calculation function in the SmartPad calculator;

FIG. 53 is a flow chart describing the logic of theGenerate-Vertical-Calculation function in the SmartPad calculator;

FIGS. 54A, 54B, and 54C together are a flow chart describing the logicof the Generate-Copy-Calculation function in the SmartPad calculator;

FIGS. 55A and 55B together are a flow chart describing the logic of theFind-Object-To-Add-To-Horizontal-Calculation function in the SmartPadcalculator;

FIGS. 56A, 56B, and 56C together are a flow chart describing the logicof the Find-Object-To-Add-To-Vertical-Calculation function in theSmartPad calculator;

FIG. 57 is a flow chart describing the logic of theCreate-Calculation-Result function in the SmartPad calculator; and

FIGS. 58A and 58B together are a flow chart describing the logic of theCalculate function in the SmartPad calculator.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description of the preferred embodiment, reference ismade to the accompanying drawings which form a part hereof, and in whichis shown by way of illustration a specific embodiment in which theinvention may be practiced. It is to be understood that otherembodiments may be utilized and structural changes may be made withoutdeparting from the scope of the present invention.

1. PURPOSE

FIG. 1 illustrates the packaging for a SmartPad calculator according tothe present invention. The SmartPad calculator 10 consists of a display12 simulating a sheet of paper and a stylus 14 simulating a pen. Theuser writes a calculation on the SmartPad calculator as though it was apiece of scratch paper. The SmartPad calculator works like scratchpaper, but does the calculations automatically rather than requiring theuser to perform manual calculations. The SmartPad calculator combinesthe best features of manual calculations with the power and accuracy ofelectronic calculators. The SmartPad calculator uses handwritingrecognition to determine what the calculation is, performs thecalculation, and then displays the result at an appropriate location(below the line or after the equal sign).

2. HARDWARE

FIG. 2 illustrates the electronic components comprising the presentinvention. The SmartPad calculator preferably comprises a flat hardwarepackage consisting of a screen display 12 combined with digitizer inputsurface, a stylus 14, a processor 16 with access to read-only 18 andread-write memory 20, and an electrical power source 22. Optionally, theSmartPad calculator may have removable memory 24, a data interface 26,and a printer 28.

Package

The SmartPad calculator works best in a device that lies flat, like anotebook. This is because the user writes directly onto the surface ofthe screen. Typically, the user places the SmartPad calculator on adesk, or holds the SmartPad calculator in their hands or on their lap.The SmartPad calculator can be as small as a credit card or as large asa pad of accounting paper. The SmartPad calculator works in anyorientation, and the orientation can be changed while in use.

Screen

The SmartPad calculator's screen takes up nearly all of one side of thepackage. In fact, to the user, the SmartPad calculator is not much morethan a screen. Ideally, the screen makes the SmartPad calculator looklike a sheet of paper. However, current low-power electronic displaytechnology (e.g., LCDs) is not really as bright as paper, although thisshould be corrected as display technology advances.

The SmartPad calculator's screen is preferably a rectangular grid ofindividually addressable pixels, with a resolution of at least 30 pixelsper inch. Ideally, the pixels are square. Since the SmartPad calculatorscreen is combined with a digitizer input surface, it is capable ofsensing which pixel is under the stylus tip when the stylus is incontact with the screen's surface.

Stylus

The user "writes" calculations on the SmartPad screen with a stylus. Thestylus feels like a pen or pencil, but dispenses "electronic ink" andnot real ink. The user writes directly onto the screen, and theelectronic ink is displayed directly underneath the tip of the pen asthe user writes.

The screen surface makes stylus writing feel like writing with a realpen on real paper, so the user writes naturally. Preferably the SmartPadcalculator has a dock to hold the stylus when not in use. Additionally,in the preferred embodiment, power to the screen is turned onautomatically when the stylus is removed from the dock, and then turnedoff automatically when the stylus is returned to the dock.

Processor

The SmartPad calculator has a general-purpose microprocessor to run thesoftware that provides most of the SmartPad calculator's functionality.Low-power versions of the INTEL 80x86, MOTOROLA 680x0, as well as otherprocessors, are suitable for the SmartPad calculator. Further,special-purpose circuits or microprocessors (e.g., neural networks,ASICs, etc.) may also be used.

Read-Only Memory

The SmartPad calculator preferably uses a read-only memory (ROM) to holdthe software. The ROM may also hold fonts (character images) fordisplaying, and default character descriptions for a handwritingrecognizer.

Read-Write Memory

The SmartPad calculator preferably uses a low-power read-write memory(RAM) to hold information which changes. This may include the screenimage, calculations, and trained character descriptions for therecognizer. Some or all of the RAM may remain powered even when theSmartPad calculator is turned off, so the pad and training are not lost.

Removable Memory

The SmartPad calculator may provide removable (off-line) memory.Removable memory holds copies of the user's pads and training. Removablememory lets the user have an unlimited number of pads. It also allowspads to be backed up for safety or archived for later retrieval.Finally, removable memory provides a way to transfer pads and trainingfrom one SmartPad calculator to another.

Current technology offers two kinds of removable memory that are mostsuitable for the SmartPad calculator: magnetic cards and floppy disks.Compared to floppy disks, magnetic cards are smaller, lighter, and useless power. However, floppy disks can be read by personal computers.

Data Interface

The SmartPad calculator may provide a data interface. The data interfaceallows information to be transmitted to and received from other devicessupporting the same interface. The other device might be anotherSmartPad calculator, a computer, a printer, or a fax machine. Bytransferring information to another SmartPad calculator, the user canshare pads with another user, or move pads to larger or more portablehardware. By transferring information to a computer, the user can use apersonal computer version of the SmartPad calculator, which may providekeyboard input of numbers, faster calculations, more printing options,and exporting to spreadsheets. By transferring information to a printeror fax machine, calculations can be printed or sent to another person.

The SmartPad calculator's data interface might be an infraredtransceiver, a serial (RS-232) cable, or a direct telephone connection.An infrared transceiver uses little power and takes little space, buttypically is only compatible with other SmartPad calculator devices inthe immediate vicinity. A serial interface uses more power, takes morespace, and is not well standardized. However, a serial interfaceconnects to computers, printers, and modems. Finally, a direct telephoneconnection may connect to fax machines and, through modems, tocomputers.

Printer

The SmartPad calculator may provide a built-in printer to copy thescreen or other data onto paper. Printing gives the user a permanentrecord of a calculation, suitable for backup, archiving, or sending toanother person. Although plain paper printing is preferable, plain paperprinters are bigger, more expensive, and use much power. Thermalprinting onto a special waxed paper does not have these problems, butthe paper is harder to get, is easily marked, and fades. Alternatively,the user can print through the data interface, as described above.

Power

Of course, the SmartPad calculator requires electrical power. Typically,this is either AC (plugging into a standard electrical outlet) or DC(installing a battery inside the package) or both. The battery may beeither a common watch or calculator battery, a standard radio battery,or a rechargeable NiCad battery. With an AC supply, batteries are stilluseful to keep RAM powered while the SmartPad calculator is not pluggedin, or to provide portability. Unless the batteries are rechargeable,they should keep the SmartPad calculator powered a long time, tominimize the frequency of battery changes. The SmartPad calculator alsohas an internal power "reservoir" (capacitor) which keeps RAM poweredwhile the battery is being changed or charged.

4. PLATFORMS

The SmartPad calculator runs on dedicated devices, pen-based computers,and even on regular computers.

Dedicated Devices

Unlike most software applications, the SmartPad calculator is quiteuseful running on a dedicated device, i.e., one that only runs theSmartPad calculator software. The smallest version of the SmartPadcalculator may be the size of a credit card, and have a 64×128 pixeldisplay, a small stylus, 64K ROM, and 4K RAM. It most likely has noremovable memory, data interface, or printer. It uses watch batteries,which last at least a year. The pad may only have one small page. On theother hand, by reducing the number of components, the cost can be keptlow, for example, $25.

An intermediate version of the SmartPad calculator may be a bit narrowerthan a letter envelope, and have a 256×512 pixel display, a normalstylus, 128K ROM, and 64K RAM. Such a version may provide some way toprint calculations, e.g., a thermal printer with an infrared transceiverinterface. It may also use radio batteries, which last a few months. Thecost of a such a unit could be kept as low as $150.

The largest version of the SmartPad calculator may be the size of asheet of paper (maybe larger), and have a 768×512 pixel display, 256KROM, and 128K RAM. It may provide some kind of computer interface andhave an option for a built-in printer. It may also use AC power and haverechargeable batteries for portability. It could include other features,such as note-taking, time-management, and phone directory. The cost of asuch a unit could be kept as low as $500.

Pen-Based Computers

The SmartPad calculator is a useful application on pen-based computers.Pen-based computers are well known in the art, including such examplesas Tandy Corporation's Grid unit, NCR Corporation's System 3125, andApple Computer's Newton Personal Digital Assistant (PDA).

Pen-based computers use a stylus as their primary input device.Pen-based computers can be as small as a paperback novel or as large asa pad of letter-size paper. They are useful for filling out forms,finding and viewing information stored electronically, and taking notes.

Typically, pen-based computers operate under the control of aspecialized operating system, for example, Microsoft Corporation's PenWindows™, GO Corporation's PenPoint™, and Communications IntelligenceCorporation's PenDOS™. The means for recognizing handwritten input isembedded in the operating system, and as such, applications need littleor no modification to run under such operating systems. Mouse andkeyboard operations are usually emulated by the movement of or tappingof the stylus on the writing surface of the computer.

With the SmartPad calculator, pen-based computers are also useful forsimple-to-moderately complex calculations that otherwise require acalculator or spreadsheet. Pen-based computers and their operatingsystems might "bundle" a minimal version of the SmartPad calculator.Users could buy a more sophisticated version separately. On a pen-basedcomputer, the SmartPad calculator uses the computer's availablehardware, including removable memory, data interfaces, and printers.

Regular Computers

The SmartPad calculator may also be useful on regular computers,especially those with graphical user interfaces. On a regular computer,the data from a SmartPad calculator is easily printed, backed up,archived, exported to spreadsheets, or sent to other computers orSmartPad calculator devices. The SmartPad calculator on a regularcomputer is also an attractive alternative to spreadsheets for users.The disadvantage of the SmartPad calculator on a regular computer is thelack of handwriting recognition software and hardware. This requires auser interface that is significantly different, using the keyboard fornumber and operator entry and the mouse or cursor keys for placement,selection, and command selection.

5. USER INTERFACE

The SmartPad calculator's user interface is characterized by "scratchpaper" and "notepad" metaphors, along with real-time handwriting andcalculation recognition. In addition, the user trains the SmartPadcalculator with a "learning palette".

Scratch Paper Metaphor

FIG. 3 illustrates the scratch paper metaphor as presented to the useron the screen 12 of the SmartPad calculator. When a new page is created,it is completely blank. The user writes out calculations in a naturalmanner, as if they were being done manually. Moreover, calculations canbe written horizontally or vertically.

Unlike real scratch paper, the SmartPad calculator computes the result,i.e., 6.0 and 24.98, automatically as soon as an equals sign ("=") isdrawn to the right of the numbers. For vertical calculations, theSmartPad calculator computes the result, i.e., 30.98 and 2.17, as soonas a "result line" is drawn underneath the numbers.

As with real scratch paper, the user can write calculations anywhere onthe screen, and in any orientation. In addition, numbers and symbols canbe written in any reasonable size. Unlike scratch paper, the SmartPadcalculator might allow screens to be of arbitrary size, so the usernever runs out of space to add more calculations (until out of RAM orvirtual memory).

Notepad Metaphor

The SmartPad calculator presents a notepad metaphor on the screen 12,which means that it operates like a pad of paper. Each piece of paper isone page of the pad. The user can create a new blank page or throw awayan old used one. The user can have more than one page in the pad filledwith calculations, but the screen only shows one page at a time. Unlikereal pads, the SmartPad calculator allows an arbitrary number of pages(until out of RAM). To identify pages, each page is numbered, i.e., "4"in FIG. 3, and the user may write in a title, i.e., "order for John Doe"in FIG. 3. The user can switch pages at any time; when switching, theuser can go from one page to the next in sequence, or choose a page froma table of contents showing all the titles and page numbers.

Real-Time Recognition

The SmartPad calculator uses handwriting recognition technology toidentify the digits and symbols written by the user. The SmartPadcalculator recognizes characters in real time (as soon as the userwrites them). Once recognized, the "electronic ink" is replaced on thescreen by a well-formed character of approximately the same size; thisgives the user immediate feedback on the accuracy of the recognition.

In the SmartPad calculator, character recognition is fast enough to keepup with a user writing as quickly as humanly possible. Characterrecognition is also a background process, so the writing of the nextcharacter never has to wait until the previous character is recognized.

After character recognition has occurred and the character identified asa digit, the SmartPad calculator uses a "number recognizer" to identifywhich number the digit belongs to. The number recognizer looks for anyother numbers that are close to the new digit; if one is found, theSmartPad calculator incorporates the digit into the number. If no othernumber is close, the SmartPad calculator creates a new single-digitnumber. Once recognized as part of a number, the SmartPad calculatormakes all characters the same size, aligns them horizontally, andadjusts the spacing between them. These adjustments show the user thatthe number was properly recognized, and helps keep the user's writing ontrack.

As soon as the SmartPad calculator recognizes a calculating symbol (suchas the equal sign or the result line), the SmartPad calculator uses a"calculation recognizer" to identify the numbers and operators that arepart of the calculation. Once a calculation is recognized, all of itscharacters are made uniform in size, aligned horizontally and/orvertically, and spaced evenly. Anything nearby but not a part of thecalculation is pushed away. This gives the user immediate feedback onthe accuracy of the calculation recognizer. Thereafter, the SmartPadcalculator does the calculation and displays the result in theappropriate place.

Training Palette

FIG. 4 illustrates a "training palette" 30 as presented to the user onthe screen 12 of the SmartPad calculator. This palette is displayed atthe top of the screen 12, and shows each of the characters and gestureswhich the SmartPad calculator recognizes. If the SmartPad calculatorincorrectly recognizes any character or gesture, the user may press thestylus on the correct one in the palette 30, and the SmartPad calculatormakes the appropriate change to the page. The SmartPad calculator alsoremembers how the user wrote that character, so next time it isrecognized correctly. This "trains" the SmartPad calculator to recognizethe user's handwriting, no matter how bad it is. To minimize therecognition error rate, users should spend a little time training a newSmartPad calculator device. Once trained, it is faster to makecorrections by rewriting the character better.

6. NUMBERS

Numbers are obviously an important part of the SmartPad calculator.Numbers consist primarily of digits. However, there is more to numbersthan just digits, and there are different kinds of numbers.

Number Entry

The user normally enters a number into the SmartPad calculator bywriting its digits in sequence from left to right on the screen. TheSmartPad calculator's number recognizer does not require this particularorder, but does require that digits be placed on the page in reasonableproximity to each other, mostly horizontal, with higher-order digits tothe left of lower-order digits. The digits can be any size (withinreason), but all the digits of a number should be roughly the same size.A simpler alternative number recognizer would just add digits to thenumber in the order written, but then the user could not enter digitsout of order. The SmartPad calculator may limit the number of digits ina number. If there is a limit, it is not less than nine digits in thesmallest version of the SmartPad calculator, and is higher in otherversions.

Besides digits, the user can write decimal points and thousandseparators (periods and commas in the U.S.). The SmartPad calculatorrecognizes and displays well-formed versions of these characters as theyare written. The user is not required to enter a decimal point unlessthe number contains non-zero fraction digits; the SmartPad calculatoradds the decimal point and zero fraction digits if appropriate. The useris also not required to enter thousand separators; the SmartPadcalculator adds them to the number when it is fully recognized (afterthe last digit is entered).

Number Alignment

Numbers in the SmartPad calculator are subject to baseline, size, andspacing alignments. Alignment occurs as the number is recognized.Baseline alignment means that the bottoms of the digits are aligned to asingle imaginary (e.g., invisible) horizontal line. Size alignment meansthat all the digits of the number have the same size. In other words,the widths and heights of the digits are identical. Finally, spacingalignment means that the digits are spaced uniformly. Numbers must alsobe aligned to other parts of a calculation, but the SmartPad calculatordoes this when the calculation is recognized.

Zero Fill

To simplify number entry, the SmartPad calculator provides an automaticzero-fill function in certain situations; sometimes, the user may haveto invoke the zero-fill function by writing a dash or other specialcharacters. Zero-fill means that one or more zeroes are appended to anumber without the user writing them. When writing a horizontalcalculation (or the first number of a vertical calculation), thezero-fill function is less automatic. If the number includes athousand's separator, the zero-fill function adds zeroes until there arethree digits after the separator. For example, writing "3,2-" results in"3,200". After a decimal point, the zero-fill function adds enoughzeroes to match the number of fraction digits in the last numberentered; the default is two fraction digits. For example, writing "4.-"could result in "4.00". In other situations, the zero-fill function hasless meaning, and results in adding a single zero. For example, writing"5-" would result in "50".

In vertical arrangements of numbers, the zero-fill function is moreautomatic. In that case, the vertical placement of the digits determineshow many zeroes are added. The SmartPad calculator keeps unit's digitsaligned vertically, and keeps the same number of fraction digits in eachnumber. Thus, if the user first writes "2,845,605.15", and then belowthat writes "79-", with the "79" underneath the "84" in the firstnumber, the SmartPad calculator produces "790,000.00".

After filling out a number with zeroes, the zero digits are like anyother digit written by the user; therefore, the user can change any zeroby writing over it.

Whole Numbers

Whole numbers are integers that are not negative. Whole numbers include"0", "1", "52", and "3,195,384". The user enters whole numbers by simplywriting the digits, optionally including thousand separators. Wholenumbers are displayed as written, except that thousand separators areautomatically included after the number is completed.

Real Numbers

Real numbers are numbers that include a fraction part. Real numbersinclude "6.4", "7,498.21", "0.000", and "5.". The user enters realnumbers by writing the digits and the decimal point, optionallyincluding thousand separators. Real numbers are displayed as written,except that thousand separators are automatically included.

Negative Numbers

The user does not enter negative numbers, but instead writes symbolsthat cause a number to be negated in a calculation. There are two waysto negate a number: (1) either write a minus sign in front of it, or (2)enclose it in parentheses. The SmartPad calculator displays the minussign or parentheses as the user writes them. When the result of acalculation is a negative number, the SmartPad calculator displays it inthe same way the user has negated numbers elsewhere; the default is touse the minus sign.

Currencies

Numbers representing monetary amounts (currencies) are the most commontype of number appearing in everyday calculations. In the U.S.,currencies are normally prefaced with a dollar sign "$" and have twofraction digits (e.g., $211.65). The SmartPad calculator is capable ofrecognizing dollar signs and carrying them through calculations,automatically prefixing dollar signs to the results where appropriate.Versions of the SmartPad calculator for other countries would recognizethe appropriate currency symbol and display the results in the rightformat.

Fractions

Users might sometimes want to enter fractions instead of the decimalequivalent. Often, fractions are more accurate. The SmartPad calculatorsupports fractions, and allows the user to write them in a naturalmanner.

Calculations with fractions retain full accuracy, so the results mayalso contain fractions. The SmartPad calculator does not convertfractions to decimal, as this may produce round-off error. Fractionresults are reduced (e.g., "2/4" becomes "1/2").

Other Numbers

There are many other kinds of numbers that users work with and that aresupported by the SmartPad calculator. These include measurements (suchas feet and inches, for example: 9'10"), dates (such as 5/18/92), times(such as 11:31 AM), scientific notation (such as 4.123×10⁸), complexnumbers (such as 12+5i), numbers in non-decimal bases (such as 274₈ or4E7A₁₆), and even matrices. These numbers require special algorithms forperforming arithmetic operations. Some kinds of numbers would only besupported in specialized versions of the SmartPad calculator (forexample, an engineer's version).

Counters

FIG. 5 illustrates counter functions 32-38 as presented to the user onthe screen 12 of the SmartPad calculator. Sometimes the user requiresitem counts as input to a calculation, for example, adding up the valueof some change by counting the numbers of pennies, nickels, dimes, andquarters. Typically, the user sorts the coins by type, and then countseach type separately. The SmartPad "counters" 32-38 are icons that keeptrack of how many times they are tapped by the stylus. Counters 32-38start at zero, so that each time the stylus touches the screen 12 of acounter 32-38, its value is increased by one. There is also a way todecrease its value by one, in case the tap was a mistake. Ideally, eachchange in value would be accompanied by audio feedback. The value of thecounter 32-38 is used in calculation like any other number. Withcounters 32-38, the user does not have to sort the coins. Instead, aseparate counter 32-38 is created for each type of coin, and the coinsare then processed in any order; wherein the appropriate counter 32-38is tapped once for each coin. While this is going on, the SmartPadcalculator can be computing a running total.

7. OPERATORS

Operators are the parts of calculations that are not numbers. Operatorsspecify the operations to perform on the numbers. Most operators aresingle symbols, such as the plus sign "+". As with digits, the SmartPadcalculator uses handwriting recognition technology to determine whichoperator is written on the screen 12.

Arithmetic Operators

The SmartPad calculator supports the four basic arithmetic operators:add, subtract, multiply, and divide. Addition is represented by the plussign "+", and subtraction by the minus sign "-". Multiplication isrepresented by the letter "×" or a raised period ·. Multiplication isalso represented by the at sign "@", as in "5 @ $1.05=$5.25". Divisionis represented by a slash "/" or a divide sign "÷". In addition, theSmartPad calculator supports percentages, represented by the percentsign "%". Percentages are usually multiplied, added, or subtracted, asin the following: "3.00×6%=0.18", "3.00+6%=3.18", and "3.00-6%=2.82".Numbers may be divided by percentages as well, as in "3.00÷6%=50.00",which tells you that 3 is 6% of 50.

All but the smallest version of the SmartPad calculator supports thesquare root operation. However, the natural representation of squareroot is not a simple character; instead, the user writes: ##EQU1##

A line is drawn over the entire expression whose value is input to thesquare root operation. A simplified version of the SmartPad calculatormight use parentheses instead of the line, as in "√(30+19)", but this isless natural.

The SmartPad also supports the power operation (e.g., squaring). Unlikeother operations, there is no symbol for the power operation; instead,it is represented by making the power a superscript, as in "15² ". Tosupport this representation, the number recognizer must be careful notto consider the exponent part of the base. The calculation recognizerthen can identify the power operation based on the relative placement ofthe two numbers.

Ordering Operators

Ordering operators control the order in which operations are performed.Usually, operations that are performed first are enclosed inparentheses, as in:

5×(3+4)=35

Besides parentheses, the SmartPad calculator also supports brackets "[]"and braces "{}". Ordering operators only apply to horizontalcalculations; vertical calculations are always performed from top tobottom.

Special Functions

Specialized versions of the SmartPad calculator may support a number ofmathematical functions. Business users could use time-value-of-moneyfunctions, which compute present and future value, loan payments, etc.Accountants could use depreciation and amortization functions. Engineerscould use trigonometric and logarithmic functions. Some functions havegraphic representations (e.g., integration), but most are represented byletters and parentheses, as in:

sin(25°)

To handle functions like this, the handwriting recognizer must recognizeletters as well as digits and operators. An alternative embodiment wouldrecognize the entire function name (e.g., "sin"). This may be moreaccurate, and will allow names to be written in script instead ofindividually printed letters.

Conditionals

The SmartPad calculator keeps track of the logical structure of thecalculations on a page. This allows the user to go back and fix anymistakes without redoing the entire calculation. This also means theSmartPad calculator page is a template in which the user can enter inputvalues and view output results. To be useful, this feature requiresconditional functions, which are functions that use different formulasdepending on one or more conditions. For example, computing the absolutevalue requires a conditional function of the form:

if xxx<0 then -xxx else xxx

where the second and third "xxx" are copies of the first "xxx." Torepresent conditional functions, those with math training use braces toenclose a vertically arranged list of conditions as illustrated in FIG.6.

However, recognizing the layout in FIG. 6 may be challenging.Alternatively, the SmartPad calculator may use the spreadsheetrepresentation (an "if" function) as illustrated in FIG. 7.

Another common conditional function is table lookup. The SmartPadcalculator supports various kinds of table lookup. The simplest is toselect a single value from a list of values, given the index of thevalue in the list. A more complex lookup interpolates on a table offunction arguments and results. The table lookup functions arerepresented by the name of the table, as in:

tax ($20,000)

To name a table, the user creates the table and writes the name on top.

Constants

A version of the SmartPad calculator supporting trigonometric andlogarithmic functions is not complete without the mathematical constants"π" and "e". Besides these constants, the SmartPad calculator supportsmore obscure scientific constants, as well as metric conversions.

Other Operators

In addition to the operators listed above, the SmartPad calculator maysupport numerical computation operations, including but not limited to,arbitrary-precision arithmetic, complex numbers, combinatorialfunctions, integer functions, matrix operations, root finding, functionfitting, Fourier transforms, numerical integration, numericaldifferentiation, function minimization, and linear programming. TheSmartPad calculator may also recognize symbolic computation operators,including but not limited to, equation solving, symbolic integration,symbolic differentiation, power series, limits, algebraic operations,polynomial expansion, factorization, simplification, matrix operations,tensor operations, list operations, and string operations.

8. CALCULATIONS

The heart of the SmartPad calculator's functionality is the calculation.There are actually different types of calculations, each of which worksdifferently. The main types are "horizontal", "vertical", and "copy."

Horizontal Calculations

In horizontal calculations, the numbers and operators are arrangedhorizontally:

25×(3,809+265)-14,266=87,584

Horizontal calculations are identified by the equals sign "=". As soonas the user writes an equals sign, the SmartPad calculator invokes thehorizontal calculation recognizer to identify which numbers and symbolsare part of the calculation, and their order within it. To identifycomponents of the calculation, the recognizer looks for objects that arenot part of another calculation, are arranged mostly horizontal, arereasonably close to each other, and include the equal sign on the right;results of other calculations are included if they have not already beenmade part of a separate calculation. The ordering of the objects withinthe calculation is determined by their horizontal placement, with theleftmost object being first.

A simpler alternative recognizer just uses the objects written since thelast calculation, in the order written, but this does not work if theuser writes out of sequence. This type of recognizer is appropriate onlyfor a minimal version of the SmartPad calculator in which the logic ofthe calculation is not saved.

Once a horizontal calculation is recognized, the SmartPad calculatoraligns the baselines, sizes, and spacing of the objects in thecalculation. Then, the SmartPad calculator does the calculation anddisplays the result. When evaluating a horizontal calculation, the usualrules of operator precedence are applied, except that anything withinparentheses is done first. Most operators are infix operators, meaningthat the operator is between the numbers it operates on. The result ofthe calculation is a number that is displayed like other numbers; it isaligned with the rest of the calculation but is to the right of theequal sign. If necessary, the SmartPad calculator moves calculationsaround to make room for the result or to maintain other alignments.

In some cases, horizontal calculations are too long to fit across thepage (or screen). On real scratch paper, these calculations are "broken"into two or more lines:

25×(3,809+265)

-14,266=87,584

The SmartPad calculator's horizontal calculation recognizer is capableof identifying calculations like this.

Vertical Calculations

Vertical calculations are calculations in which the numbers andoperators are arranged vertically:

    ______________________________________                                                     3,809                                                                         +265                                                                          ×25                                                                     -14,266                                                                       87,584                                                           ______________________________________                                    

Vertical calculations are identified by the result line, which is theline underneath the numbers. As soon as the user draws the result line,the SmartPad calculator invokes a vertical calculation recognizer toidentify which numbers and symbols are part of the calculation, andtheir order within it. To identify components of the calculation, thevertical calculation recognizer first looks for objects immediatelyabove the result line that are not part of another calculation, aremostly horizontal, and are reasonably close to each other. These objectsare considered part of the bottom line of the vertical calculation.After that, the next to bottom line is identified by looking for objectsimmediately above the bottom line meeting the same conditions. Thisprocess repeats until the top line is found, which means there is noidentifiable line above it. The ordering of the lines is determined bytheir vertical placement, with the top line being first. The ordering ofthe objects within each line is determined by their horizontalplacement, with the leftmost being first. As with horizontalcalculations, a simpler alternative recognizer could just use theobjects written since the last calculation, in the order written.

Once a vertical calculation is recognized, the SmartPad calculatoraligns the sizes and spacing of the objects in the calculation. Inaddition, the SmartPad calculator aligns the baselines of the objectswithin each line, as well as the vertical spacing between the baselinesof each line. Finally, the SmartPad calculator aligns the right sides ofthe unit's digit of the rightmost number in each line. Then, theSmartPad calculator does the calculation and displays the result. Whenevaluating a vertical calculation, operations are performed from top tobottom; there is no way to change this order, even with parentheses.Each line of the vertical calculation is supposed to contain an operatorfollowed by a number. The operation is applied to the result of thepreceding lines and the number on the same line. The operator may beleft out, and is assumed to be a plus sign in that case. The result ofthe calculation is a number that is displayed like other numbers; it isaligned with the rest of the calculation but is below the result line.Once the result is aligned, the ink of the result line is replaced witha well-formed horizontal line segment, aligned on the left and rightwith the extreme lefts and rights of the objects in the verticalcalculation. If necessary, the SmartPad calculator moves othercalculations around to make room for the result or to maintain otheralignments.

Copy Operations

FIG. 8 illustrates a copy operation as presented to the user on thescreen 12 of the SmartPad calculator. A copy operation is simply a wayto make a copy of a number without writing it twice; the copy is placedat a different location on the page, where it can be part of anothercalculation.

Copy operations are identified by an arrow that points from the originalnumber to the copy. To create a copy operation, the user draws thearrow, starting someplace near the original number and ending near thecenter of where the copied number is to be located. As soon as the arrowis drawn, the SmartPad calculator invokes the copy operation recognizerto identify the original number. The recognizer looks for a number thatis underneath the source of the arrow, or close by and in the directionopposite-to the arrow's head. Any number can be the original, even if itis part of another calculation (input or result). In particular, thenumber may be part of another copy operation, so an unlimited number ofcopies of the original can be made.

Once the original number is identified, the SmartPad calculator makes acopy and places it underneath the head of the arrow. Then, the ink ofthe arrow is replaced with a well-formed arrow starting near the edge ofthe original number and ending near the edge of the copy. The SmartPadcalculator may try to find a route for the arrow that avoids overwritingother objects on the page.

Total/Subtotal Calculations

FIG. 9 illustrates total and subtotal calculations as presented to theuser on the screen 12 of the SmartPad calculator. Accountants often usecalculations that involve subtotals and a total of the subtotals.

Total/subtotal calculations in the SmartPad calculator use regularresult lines for the subtotals and then place another result line,resulting in a double result line, underneath the grand total. Sinceregular result lines are used for vertical calculations, small changesmust be made to the way the vertical calculation recognizer works.Specifically, when there is only a single available number above theresult line, no vertical calculation is generated. Also, verticalcalculations do not extend above other result lines, even if they arenot part of another vertical calculation.

Having made these changes to the vertical calculation recognizer, theSmartPad calculator can invoke the total/subtotal calculation recognizeras soon as the user writes a double result line. This recognizersearches above the double result line for any numbers that stand alonebetween two regular result lines. All such numbers are either subtotalsof other numbers, or separate numbers that are not part of one or thesubtotals. Once these numbers are identified, they are incorporated(along with any associated operators) into the grand total calculation.The grand total calculation is evaluated like a vertical calculation,and the result is placed above the double result line. As with verticalcalculations, the numbers are aligned and the ink of the result lines isreplaced by well-formed lines that are also aligned to the calculation.

Animation

The SmartPad calculator "animates" calculations as they are beingevaluated. Preferably, calculations are animated by flashing each of itsnumbers and operators in sequence (including the result). For example,assume the user has entered the calculation of:

25×(94-57)=925

The SmartPad calculator first flashes the "25", then the "×", the "(",the "94", the "-", the "57", the ")", the "=", and finally the result"925". The animation is slow enough for the user to follow, but fastenough to finish a page in a few seconds or less. By animatingcalculations, the user knows when a calculation is being performed, andwhat numbers are involved. This feedback helps the user make sure theSmartPad calculator recognizes the right calculations at the right time.

Errors

Sometimes when doing a calculation, the SmartPad calculator finds amistake or an incomplete formula, for example:

32×7-

The kinds of mistakes the SmartPad calculator can detect include:unrecognizable character, number overflow (too many digits), division byzero, negative square root, bad number format (e.g., "34.56.2"), andimproper syntax (such as unmatched parentheses, missing operator, ormissing number). When there is a mistake, the result is undefined, so afew question marks "???" are displayed on the screen 12 instead. If theresult is itself part of another calculation, the result of thatcalculation is also undefined. However, subsequent results are displayedas dots ". . . ", so the user can easily locate the calculation with theproblem.

The SmartPad calculator might provide informative diagnostics when anerror occurs. These diagnostics indicate the kind of error, and where itis in the calculation. With this information, the user is able topinpoint and correct the error quickly.

9. LINKS

The SmartPad calculator can link calculations, which means that theresult of one calculation is input to another calculation. Once a linkis set up, the SmartPad calculator handles everything so the user doesnot have to copy results or repeat calculations. The SmartPad calculatorsaves the logical relationships, so when the user fixes a mistake orchanges an input number, all affected results are recomputedautomatically. Thus, the SmartPad calculator has the same capabilitythat makes spreadsheets so popular. There is no limit on the number oflinks, but dependency cycles are not permitted.

Inter-Calculation Links

Inter-calculation links are links between two calculations on the samepage. These links are generated automatically by the calculationrecognizer, based on placement. For example, two horizontal calculationsmay be linked:

3,809+278=4,087×3=12,261

When the second equal sign is written, the horizontal calculationrecognizer looks for numbers and operators to the left of the equal signthat are not already input to another calculation. The recognizer thusincorporates the result of the first calculation into the secondcalculation. Any calculation can link with any other calculation. Forexample, two vertical calculations may be linked:

    ______________________________________                                                     3,809                                                                         +278                                                                          4,087                                                                         ×3                                                                      12,261                                                           ______________________________________                                    

Another example, is a vertical calculation linked to two horizontalcalculations:

    ______________________________________                                                   5 × 1.25 =  6.25                                                        2 × 12.00 = 24.00                                                       30.25                                                              ______________________________________                                    

Even if two calculations are not near each other, they can be linkedindirectly through a copy operation as illustrated in FIG. 10.

Inter-Page Links

The SmartPad calculator supports links between calculations on differentpages of the same pad. This allows sophisticated users to develop verycomplex calculations. An inter-page link is similar to a copycalculation in that a number on one page is copied onto another page.Inter-page links are displayed as squares containing identifyingnumbers, as illustrated in FIGS. 11A and 11B.

The user creates an inter-page link by drawing the square and writingthe identifying number inside it. To get an unused number, a dash isdrawn inside the square, and the SmartPad calculator replaces it withthe next available identifying number. To send a number to another page,the user creates a copy operation (i.e., draws an arrow) from the numberto the link object. To get a number from another page, the user createsa copy operation from the link object. Any number of arrows can go outof a link, but only one is allowed into the link.

Inter-Application Links

On multi-tasking computers, the SmartPad calculator supportsinter-application links, which are links to values in otherapplications. For example, in a form processing application, the usermay need to calculate the value entered into a field of the form. TheSmartPad calculator can do this calculation automatically if a link isset up from the SmartPad calculator result to the form field. If thecalculation depends on values in other form fields, the user can alsoset up a link from the form to the SmartPad calculator.Inter-application links allow the user to add calculations to anyapplication. The method for creating and using inter-application linksdepends on the operating system.

10. LABELS

If a calculation must be retained for any time, the purpose of thecalculation and the meanings of the numbers should be documented. TheSmartPad calculator supports documentation with "labels." A label issimply ink left on the page. A label has no effect on the results of thecalculations. There is no limit on the number of labels on a page (untilout of RAM).

The user creates a label by writing it. There is no label recognizer.Instead, anything that is not recognizable, or drawn over an existinglabel, is considered a label. This works well for cursive writing, butmistakes occur with individually printed letters (for example, theletter "I" is recognized as the digit "1"). To restore the original inkof a recognized character, the user taps the "label" command in thetitle area (discussed in more detail below in the section entitled"Pages"). To minimize these mistakes, the SmartPad calculator avoidsrecognizing characters in the midst of many unrecognizable characters.

11. SELECTIONS

Most of the time, a user of the SmartPad calculator simply writes ordraws things and the SmartPad calculator recognizes them. However, a fewactions (such as moving objects) require that objects be selected first.Once a selection is made, the action is performed with the stylus.Instead of dispensing ink, a "cursor" appears on the screen 12 beneaththe tip of the stylus, and the cursor acts as if it is attached to thestylus. Any number of objects can be selected; initially, none areselected. Objects are deselected when the user goes back to writing ordrawing.

To make a selection, the user simply draws a "lasso" around the objectsto be selected, as illustrated in FIG. 12A. The lasso is either a largeclosed figure, or a small figure resembling a zero except for the"handle" of the lasso. The SmartPad calculator selects an object ifnearly all of its interior is inside the lasso. Selections arecumulative, so more objects are added to the selection by lassoing them.To deselect objects, the user lassoes them. As long as no unselectedobjects are inside the lasso, the objects in the lasso are removed fromthe selection. To deselect everything, the user taps the stylusanywhere.

The SmartPad calculator flashes the selected objects so the user can seewhich ones they are. In addition, the SmartPad calculator displays fourhandles (square dots) in a rectangular arrangement around the selectedobjects, as illustrated in FIG. 12B. The handles are used to change thesizes of the selected objects.

12. CORRECTIONS

Everybody makes mistakes, so it should be easy to find mistakes and fixthem. Most users of the SmartPad calculator are probably notsophisticated in mathematics, and sophisticated users may developcomplex calculations. In either case, finding and fixing mistakes easilyis important. For finding mistakes, the SmartPad calculator isinherently easy because the entire calculation is displayed and nothingis hidden from the user. This is better than spreadsheets, which displayall the numbers but leave the formulas invisible (other than theselected one). For correcting mistakes, the SmartPad calculator providesmany options, most of which have a simple interface consistent with thescratch paper metaphor.

Undo Function

Most of the time, the user notices the mistake soon after making it. Inthese situations, the "undo" command makes correction easy. The user canundo virtually anything; even an "undo" can be undone with a "redo"command. The SmartPad calculator allows more than one past action to beundone. The number of undo-able past actions depends on the amount ofRAM available. The undo and redo commands are always available. The userinvokes these commands by tapping the mnemonics or icons displayed inthe title area.

Training Palette

The "training palette" 30 of FIG. 4 is used to correct mistakes by thehandwriting recognizer, but not mistakes by the user. This allows therecognizer to learn from its mistakes and adapt to the user'shandwriting. The training palette 30 is displayed in the title area ofthe screen 12 (instead of the title) when the user invokes the "learn"command. The palette 30 is a rectangular area with one box for eachsymbol (digit, operator, gesture, etc.) recognized by the handwritingrecognizer.

After writing something not recognized correctly, the user taps thestylus on the box with the correct symbol. The SmartPad calculatorchanges the symbol recognized to the symbol tapped and then adjusts itsrecognition process to do a better job next time.

People usually write too fast to make corrections immediately, so therecognition error might not be noticed until after a few more charactersare written. In that case, the user must select the incorrect symbol andthen tap the correct one in the palette. The SmartPad calculator makesthe correction, but the training feature would not be invoked if theoriginal ink has been purged from RAM. The SmartPad calculator normallyretains the ink for the last several symbols.

Users should "train" the SmartPad calculator when they first receive it,before trying their first real calculation. This avoids the distractionof correcting mistakes while thinking about the calculation. Trainingalso ensures good recognition for symbols that are not used often. Totrain the SmartPad calculator, the user writes each symbol a few timesand then corrects any that are recognized wrong. The training palette 30shows the user which symbols to write. Alternatively, the SmartPadcalculator may have a training mode in which it prompts the user towrite the symbols. Training could also be part of a tutorial or a gamethat the SmartPad calculator provides for new users. When in trainingmode, there is also a way to "untrain" the SmartPad calculator, forsomeone using another person's SmartPad calculator.

Corrections to Numbers

Most user mistakes are incorrect numbers, especially when the number iscopied from paper. Typical mistakes are incorrect digits, digits leftout, and digits repeated, but sometimes the number is completely wrong.After fixing the number, the SmartPad calculator repeats all affectedcalculations and updates the screen 12 with the new results.

FIGS. 13A, 13B, and 13C illustrate how corrections are made on thescreen 12 of the SmartPad calculator by overwriting the number. Assume,for example, that the number "7215" in FIG. 13A should be the number"7265". To fix an incorrect digit, the user writes the correct digitover the incorrect one, as illustrated in FIG. 13B. When writing on topof a character that already exists, the SmartPad calculator replaces theexisting character with the new one, i.e., the number "7265", asillustrated in FIG. 13C. Sometimes users try to "transform" theincorrect digit to the correct one. For example, to change a "1" to a"4" the user might skip writing the vertical line, since the "1" alreadyis a vertical line. The SmartPad calculator also recognizes correctionsof this type.

FIGS. 14A, 14B, 14C, 14D and 14E illustrate how corrections are made onthe screen 12 of the SmartPad calculator by inserting digits in thenumber. Assume, for example, that the number "725" in FIG. 14A should bethe number "7265". To insert a digit that was left out, the number mustfirst be "split open". In the SmartPad calculator, this is done bywriting a caret " " where the new digit is to be inserted, asillustrated in FIG. 14B. The caret is a standard proofing mark toindicate insertion. Once the caret is recognized, the SmartPadcalculator inserts a blank space where the caret was written, asillustrated in FIG. 14C. The user then writes the digit over the blankspace as illustrated in FIG. 14D. The SmartPad calculator replaces theblank space with digit, as illustrated in FIG. 14E. To insert more thanone digit, the user writes one caret for each digit and then writes thedigits.

FIGS. 15A, 15B and 15C illustrate how corrections are made on the screen12 of the SmartPad calculator by deleting digits from the numbers.Assume, for example, that the number "72165" in FIG. 15A should be thenumber "7265". To remove extra digits, the user draws a "delete" symbolover them, as illustrated in FIG. 15B. The delete symbol is a linethrough the digits ending with a pigtail. This is a standard proofingmark for deletion. Once recognized, the SmartPad calculator removes thedigits under the delete symbol and closes up the blank space, asillustrated in FIG. 15C. Any number of digits can be removed with thedelete symbol.

FIGS. 16A, 16B and 16C illustrate how corrections are made on the screen12 of the SmartPad calculator by interchanging digits in the numbers.Assume, for example, that the number "7625" in FIG. 16A should be thenumber "7265". To interchange the digits, the user draws a "interchange"symbol over them, as illustrated in FIG. 16B. The interchange symbol isa "sine wave" separating the digits. This is a standard proofing markfor interchanging. Once recognized, the SmartPad calculator swaps thedigits under the interchange symbol, as illustrated in FIG. 16C.

FIGS. 17A, 17B, 17C, 17D and 17E illustrate how corrections are made onthe screen 12 of the SmartPad calculator by erasing digits from thenumbers and replacing them with new digits. Assume, for example, thatthe number "72165" in FIG. 17A should be the number "71625". To replacemost or all of the number, the user draws an erase symbol over thedigits, as illustrated in FIG. 17B. The erase symbol is a line back andforth through the digits to be erased, as if they were being scratchedout. Unlike deletion, erasure does not close up the blank spaces, butinstead leaves the area blank so new digits can be written, asillustrated in FIG. 17C. The user then writes the digits over the blankspace, as illustrated in FIG. 17D. The SmartPad calculator replaces theblank spaces with the digits written, as illustrated in FIG. 17E.Besides correcting numbers entered incorrectly, this method is used toexperiment with different inputs to the calculations.

Corrections To Calculations

Sometimes the numbers are right but the calculation is wrong. Typicalmistakes are incorrect operators, leaving something out, doing too much,doing the wrong thing, and doing things in the wrong order. After fixingthe calculation, the SmartPad calculator computes the result, repeatsany other affected calculations, and updates the display with the newresults.

FIGS. 18A, 18B and 18C illustrate how corrections are made on the screen12 of the SmartPad calculator by overwriting the operator. Assume, forexample, that the addition operator "+"in FIG. 18A should be themultiplication operator "×". To fix an incorrect operator, the userwrites the correct operator over the incorrect one, as illustrated inFIG. 18B. When writing on top of an operator that already exists, theSmartPad calculator replaces the existing operator with the new one,i.e., the multiplication operator "×", as illustrated in FIG. 18C.Sometimes users try to "transform" the incorrect operator to the correctone. For example, to change a the user might skip writing the horizontalline, since the "-" already is a horizontal line. The SmartPadcalculator also recognizes corrections of this type.

FIGS. 19A, 19B, 19C, 19D, 19E and 19F illustrate how corrections aremade on the screen 12 of the SmartPad calculator by inserting operatorsand numbers. Assume, for example, that the calculation "5+6=" in FIG.19A should be the calculation "5+6+7=". To add something left out of acalculation, the user first inserts blank space into the calculation,then writes in the new operators and numbers. The caret (which insertsdigits in numbers) is usually not appropriate since it does onecharacter at a time. Instead, the user should select and move one partof the calculation away from the other, as illustrated in FIGS. 19B,19C, and 19D. Once enough space is blank, the user writes in the newpart of the calculation, as illustrated in FIG. 19E. The SmartPadcalculator recognizes when something new is added to a calculation byplacement. If the new objects are written inside (or just before) thearea occupied by the calculation, the SmartPad calculator inserts theminto the calculation, as illustrated in FIGS. 19E and 19F.

FIGS. 20A, 20B and 20C illustrate how corrections are made on the screen12 of the SmartPad calculator by deleting operators and numbers from thecalculations. Assume, for example, that the calculation "5+6+7=" in FIG.20A should be the calculation "5+6=". To remove something from thecalculation, the user deletes it with the delete symbol, as illustratedin FIG. 20B. This is the same as removing digits from numbers. Thedelete symbol closes up the blank space, as illustrated in FIG. 20C.

FIGS. 21A, 21B, 21C, 21D and 21E illustrate how corrections are made onthe screen 12 of the SmartPad calculator by replacing operators andnumbers in the calculations. Assume, for example, that the calculation"5+6+7=" in FIG. 21A should be the calculation "5+6- 4=". To replacepart of the calculation with something new, the user first erases theincorrect part, as illustrated in FIG. 21B. The erase symbol leaves theblank space open so the user can write in the correct operators andnumbers, as illustrated in FIG. 21C. This is the same as replacingdigits in numbers, where the erase symbol is a line back and forththrough the operators and numbers to be erased, as if they were beingscratched out. Unlike deletion, the erasure does not close up the blankspace, as illustrated in FIG. 21C. Instead, the SmartPad calculatorleaves the area blank so that new operators and numbers can be writtenin, as illustrated in FIG. 21D and 21E. Besides correcting operators andnumbers entered incorrectly, this method is also used to experiment withdifferent operators and numbers in the calculations. To replace theentire calculation, the user should erase it all. To stop thecalculation without erasing it, the user can erase the result and/or thecalculating symbol, as illustrated in FIGS. 22A, 22B and 22C. Thisleaves the expression without evaluating it.

FIGS. 23A, 23B and 23C illustrate how corrections are made on the screen12 of the SmartPad calculator by interchanging operators and numbers inthe calculations. Assume, for example, that the calculation "5+6+7=" inFIG. 23A should be the calculation "5+7+6=". To interchange theoperators and numbers, the user draws a "interchange" symbol over them,as illustrated in FIG. 23B. The interchange symbol is a "sine wave"separating the operators and digits. This is a standard proofing markfor interchanging. Once recognized, the SmartPad calculator swaps theoperators and numbers under the interchange symbol, as illustrate inFIG. 23C. An alternative method of changing the order in a calculationis to select the part and then move it to the new position, asillustrated in FIGS. 24A, 24B, 24C, 24D, and 24E. There is no need tomove more than once because the SmartPad calculator moves the rest ofthe calculation to maintain spacing alignment.

Corrections to Links

Users sometimes make mistakes in the overall logic of a calculation,especially if it is even moderately complex. A logic error means thecalculations are not linked correctly. Typical mistakes include usingthe wrong result in another calculation, including an unnecessarycalculation, or forgetting to include a necessary calculation. To fixthese mistakes, the user must make changes that have a significantimpact on the calculations. These changes are a little harder to do, sothe user is less likely to accidentally make the wrong change.

Most corrections to links require that a link be broken or that a linkbe established. The method for breaking and establishing links dependson whether the link is by position, as illustrated in FIG. 25A, or bycopying, as illustrated in FIG. 25B. A position link means the result ofthe first calculation happens to be placed inside the secondcalculation. A copy link means a copy calculation is used to copy theresult of the first calculation to the second calculation.

FIGS. 26A, 26B, 26C, 26D and 26E illustrate how position links arebroken on the screen 12 of the SmartPad calculator. FIG. 26A shows ahorizontal calculation linked to a vertical calculation by a position.To break a position link, the user selects the result of the firstcalculation and moves it out of the second calculation, as illustratedby FIGS. 26B, 26C, and 26D. Since the result and its calculation muststay together, moving the result also moves its calculation, asillustrated in FIG. 26D. Any calculations with a position link into theresult's calculation also move. To make links a little hard to break,the SmartPad calculator simulates a physical link by making the resultappear to be stuck to the second calculation as the user tries to dragit out. However, if the user drags the stylus far enough away, theSmartPad calculator breaks the link. To show the user when the link isbroken, the SmartPad calculator displays the result jumping out of thesecond calculation and to the current stylus position, as illustrated inFIG. 26E.

FIGS. 27A, 27B and 27C illustrate how copy links are broken on thescreen 12 of the SmartPad calculator. FIG. 27A shows two verticalcalculations linked by a copy operation. To break a copy link, the usererases or deletes the copy (or perhaps the arrow that links the originalto the copy), as illustrated in FIGS. 27B and 27C.

Alternatively, the copy link can be broken by selecting and moving thecopy out of the second calculation (the same as breaking a positionlink), as illustrated in FIGS. 28A, 28B, 28C, 28D, and 28E. This methodof breaking the copy link does not delete the copy, so it is better ifthe copy is needed somewhere else.

FIGS. 29A, 29B, 29C, 29D and 29E illustrate how position links are madeon the screen 12 of the SmartPad calculator. FIG. 29A shows two separatecalculations, one which is a horizontal calculation and the other whichis a vertical calculation. To establish a position link between twoexisting calculations, the user selects the result of the firstcalculation and moves it into the second calculation, as illustrated inFIGS. 29B, 29C, and 29D. There is no need to make room for the result inthe second calculation, since the SmartPad calculator moves objectsaround to maintain alignment, as illustrated in FIG. 29E. As withbreaking links, the SmartPad calculator "resists" the effort toestablish the link, making it hard to do accidentally. The result of thefirst calculation stops moving when it is close to the secondcalculation, as if the result cannot enter the calculation. However, ifthe user drags the stylus far enough inside the calculation, theSmartPad calculator establishes the link and the result jumps into thecalculation.

FIGS. 30A, 30B and 30C illustrate how copy links are made on the screen12 of the SmartPad calculator. FIG. 30A illustrates two separatecalculations, both of them being vertical calculations. To establish acopy link between two existing calculations, the user creates a copycalculation from the result of the first calculation to the inside ofthe second calculation, as illustrated in FIGS. 30B and 30C.Alternatively, if there is a copy of the result that is not being used,the copy can be selected and moved into the second calculation. This isthe opposite of breaking a copy connection by moving the copy out of thesecond calculation, as described above.

Most link mistakes are fixed by a combination of breaking andestablishing links. If the wrong calculation links into anothercalculation, the user breaks the link to the wrong calculation and thenestablishes the link to the correct one. If one of the calculations isunnecessary, the user breaks any links to the unnecessary calculationand then deletes the calculation. If there are any links left danglingafter the deletion, the user deletes them or links them into theappropriate calculation. Finally, if an intermediate calculation is leftout, the user breaks the link in the main calculation where theintermediate calculation is to go and then establishes the links intoand out of the intermediate calculation. If necessary, some calculationsmay be moved first to make room for the intermediate calculation.

Corrections to Labels

FIGS. 31A, 31B, 31C and 31D illustrate how label corrections are made onthe screen 12 of the SmartPad calculator. FIG. 31A shows a label"wrong". When a label is incorrect, the user first erases it and thenwrites a new one, as illustrated in FIGS. 31B, 31C and 31D. Writing overan existing label does not fix it since the new writing is added to theoriginal; this behavior is needed so users can dot "i's" and cross"t's". To erase a label, the erase or delete symbol is drawn over thelabel.

13. ADJUSTMENTS

With scratch paper, calculations often end up laid out poorly. Forexample, part of the calculation may be too close to the edge of thepage, or two calculations may be too close together, or the wholecalculation may be too big to fit on the page. To fix these problems onscratch paper, the calculations must be erased and rewritten. With theSmartPad calculator, objects on the screen 12 are easily moved around,re-sized, rotated, and scrolled. These adjustments can be done at anytime. They have no effect on the calculations, only on the way they aredisplayed.

Moving Objects

FIGS. 32A, 32B, 32C, 32D, 32E and 32F illustrate how object positionchanges are made on the screen 12 of the SmartPad calculator. FIG. 32Ashows two objects, i.e., the numbers "628" and "91", positioned onopposite sides of screen 12. To change the position of a SmartPadcalculator object, the user first selects it with the lasso, asillustrated in FIG. 32B. Once selected, the user touches the stylus tipanywhere inside the selection, as indicated by the handles in FIG. 32C.To show that the selection can be moved, the SmartPad calculatordisplays a hand under the stylus, as illustrated in FIG. 32D. The user"drags" the stylus to a different place, as illustrated in FIG. 32E. Thehand and selection follow the stylus as if attached to it. Moving stopswhen the user lifts the stylus from the screen, as illustrated in FIG.32F. The SmartPad calculator does not allow two objects to occupy thesame space. If the user drags the stylus over another object, theselected object will stay away. However, the user can (with some"effort") move a number or operator into a horizontal or verticalcalculation; this makes the object part of the calculation. Similarly,effort is required to move an object out of a calculation.

Any number of objects can be moved in one drag by selecting all of themfirst. Each object is moved the same amount, so they maintain the samerelative positions. Since objects cannot overlap, multi-objectselections have more constraints on the places they can move to. Ifnecessary, other objects must be moved out of the way first.Multi-object selections cannot be moved into calculations unless all ofthem fit and can be part of the calculation.

Since results of horizontal and vertical calculations have fixedpositions relative to their formulas, moving a result also moves theother objects in the calculation. If any numbers in the calculation arethe results of other horizontal or vertical calculations, then allobjects in the other calculations move as well. This makes it possibleto move numerous objects by just moving one result, as illustrated inFIGS. 33A, 33B, and 33C.

With copy operations, moving the original or copy does not affect theposition of the other one. The SmartPad calculator updates the arrow topoint from the original to the copy, as illustrated in FIGS. 34A, 34B,and 34C.

Changing Size

FIGS. 35A, 35B, 35C, 35D, 35E and 35F illustrate how changes to the sizeof objects are made on the screen 12 of the SmartPad calculator. Assume,for example, that the number "628" in FIG. 35A should be enlarged forreadability. To change the size of a SmartPad calculator object, theuser first selects it with the lasso, as illustrated in FIG. 35B. TheSmartPad calculator displays four small squares (handles) around theselection, as illustrated in FIG. 35C. When the stylus tip touches ahandle, the SmartPad calculator displays a hand grabbing the handle, asillustrated in FIG. 35D. When the stylus is dragged across the screen,the hand holding the handle stays with it, except that it must remain inline with the original handle position and the handle opposite it, asillustrated in FIG. 35E. The two adjacent handles also move to maintainthe rectangular arrangement. When the user lifts the stylus, theselected object is re-sized to fit inside the new handle positions, asillustrated in FIG. 35F. The handle opposite the one grabbed does notmove, so the user chooses the direction to expand to by choosing thehandle in that direction. Objects can be increased or decreased to anysize (within reason). However, the size cannot be increased to the pointthat other objects are overwritten. Any number of objects can bere-sized in one drag by selecting all of them first. Each object isre-sized the same amount, so they maintain the same relative sizes. Inaddition, the objects move apart the same amount, so they maintain thesame relative positions and do not overlap.

All objects in a single horizontal or vertical calculation must have thesame size, except that the result can be different. Thus, changing thesize of any number or operator in a calculation causes the others tochange in size as well (except for the result). In a copy operation, theoriginal and the copy can be re-sized independently.

Changing Orientation

FIGS. 36A, 36B, 36C and 36D illustrate how changes are made to theorientation of the screen 12 of the SmartPad calculator. The SmartPadcalculator display usually has one side longer than the other, asillustrated in FIG. 36A. Horizontal calculations fit better with thelong side horizontal, while vertical calculations prefer the long sidevertical. In addition, right-handed users work better with the stylusattached on the right side, while left-handed users prefer the leftside. To accommodate these situations, the SmartPad calculator allowsany one of the four sides of the screen 12 to be "top". To change theorientation, the user draws an arrow from the middle of the current topedge to the middle of the new top edge, as illustrated in FIG. 36B. Allobjects on the page must have the same orientation, but each page in thepad has its own orientation. When the user goes to another page, thescreen 12 rotates to that page's orientation.

Scrolling

FIGS. 37A, 37B, 37C, and 37D illustrate how scrolling is performed onthe screen 12 of the SmartPad calculator. The SmartPad calculatorsupports pages of unlimited size. Rather than making objects smaller tofit in the screen 12, the user can "scroll", the screen 12 to show adifferent area of the page. Before a page can be scrolled, the user mustdraw a "compass", which is a star-like figure that remains attached tothe page, as illustrated in FIG. 37A. The SmartPad calculator respondsby displaying a formal compass icon at the indicated location on thescreen 12, as illustrated in FIG. 37B. To scroll the screen 12 over thepage, the user touches the compass with the stylus, and then drags it toa different place on the screen 12, as illustrated in FIGS. 37C and 37D.The compass (and with it the page) behaves as if attached to the stylus.The user can drag the compass anywhere on the screen 12. If morescrolling is required after the compass reaches the edge of the screen12, then a second compass can be drawn and dragged. There is no limit onthe number of compasses on a page. A compass need only be created once,since it remains until the user deletes it. Like other SmartPadcalculator objects, a compass can be selected and moved to a differentplace on the page, as long as it does not overlap other objects.

14. PAGES

Small versions of the SmartPad calculator may have only one page, butmany users want pads with multiple pages. With more than one page, userscan keep unrelated calculations on different pages. To support multiplepages, the SmartPad calculator has ways of identifying, accessing,creating, deleting, splitting, copying, merging, and sorting pages.These features also allow the user to move and copy calculations fromone page to another. The screen always displays one page at a time.

Identification and Access

The SmartPad calculator identifies pages by number and by title. TheSmartPad calculator handles page numbers automatically, wherein pagesare numbered consecutively from one. The user writes in page titles andthe SmartPad calculator treats the title as a label. The number andtitle of the current page are displayed in the "title area" at the topof the screen 12, as illustrated in FIG. 3.

The title area also displays a menu, shown as a column on the right sideof the screen 12 in FIG. 3, which includes the undo and the train orredo commands. When training, the training palette is displayed insteadof the page number and title. To change the title, the user erases thecurrent title and then writes a new one (the same as with labels). Toview another page, the user writes the new page number over the currentpage number. If the number of the other page is not known, the userdisplays page zero, which has the "table of contents".

FIG. 38 shows a table of contents, which lists pages by number andtitle. At the top of FIG. 38 is displayed the title of the pad, and onthe right is a command menu and scroll bar. The user touches the scrollbar to view pages that are not visible on the screen 12. To select apage, the user taps the page number or title once and the selected pagesare highlighted by flashing the page number. To go to any page, the usertaps the page number or title twice. Finally, the user can follow aninter-page link by tapping a link object twice. The SmartPad calculatorgoes to the other page with that link.

Creation and Deletion

Every pad has an extra blank page listed at the end of the table ofcontents, as illustrated in FIG. 38. To create a new page, the user goesto the blank page and starts writing in it. As soon as that page haswriting on it, the SmartPad calculator creates another blank page afterit. New blank pages are always available (until the SmartPad calculatoris out of RAM).

To remove a page from the pad, the user draws an erase or delete symbolover the page number in the table of contents. The SmartPad calculatorasks the user to verify that action, then removes the page from the padand discards all calculations on that page. Unless the user immediatelytaps "undo", the calculations are permanently lost.

Splitting, Copying, and Merging

FIGS. 39A, 39B, 39C, and 39D illustrate how pages can be split on thescreen 12 of the SmartPad calculator. At some point when developing alarge set of calculations, one page is not enough. To split up onepage's calculations into two pages, the user goes to the page andselects everything to place in the second page, as illustrated in FIGS.39A and 39B. Then, the user goes to the table of contents and taps the"split" command in the menu. The SmartPad calculator creates a new pageand transfers all selected objects to it, as illustrated in FIGS. 39Cand 39D. The new page is untitled and follows the original page innumbering, and the SmartPad calculator displays the new page. If thesplitting causes a result to be separated from its formula, the resultis duplicated on both pages and an inter-page link is created betweenthem, as illustrated in FIGS. 39C and 39D:

Copying calculations is similar to splitting them off. To make a copy ofsome calculations on a page, the user selects everything to copy, goesto the table of contents, and taps the "copy" command. The SmartPadcalculator copies all selected objects to the new page, but withoutremoving them from the original page. To copy an entire page, the userneed not go to the page and select everything on the page. Instead, theuser goes to the table of contents, selects the page to copy, and thentaps the "copy" command.

The opposite of splitting is merging. To merge the calculations on twoseparate pages into a single page, the user goes to the second page tomake it the current page and then goes to the table of contents. Then,the user selects the first page and taps the "merge" command. TheSmartPad calculator copies everything from the second page to the firstpage and then deletes the second page. The SmartPad calculator displaysthe first page and automatically selects everything that came from thesecond page. The user then moves the selected calculations to thedesired position relative to the calculations from the first page. Ifthere were inter-page links between the two original pages, the SmartPadcalculator replaces then with direct copy calculations. To transfercalculations from one page to another, the user first splits them off,then merges them with the other page.

Reordering

To change the order of the pages in the pad, the user first goes to thetable of contents and selects a page to move. When the stylus touchesthe selection, a "hand" is drawn to show it can be moved. The user dragsthe hand (and page, which behaves as if attached) to a new position inthe table of contents. After the stylus is lifted from the screen, theSmartPad calculator moves the selected page to the new position andautomatically renumbers every page.

15. PADS

Some SmartPad calculators support removable memory (e.g., magnetic cardsor floppy disks). With removable memory, users can save multiple pads,or multiple copies of one pad. Any number of pads can be saved (untilout of memory). To support multiple pads, the SmartPad calculator hasways of identifying, accessing, creating, deleting, splitting, copying,and merging pads. These features also allow users to transfer and copypages from one pad to another. In addition, the SmartPad calculator cantransfer pads from one device to another, and can restrict access topads.

Identification And Access

Pads are identified by a title, which is displayed in the title area ofthe table of contents, as illustrated in FIG. 38. Like page titles, theuser writes the pad title directly into the title area. Since pads areeasily copied, the SmartPad calculator also identifies pads by a copyand change number. Copy numbers are unique for each copy, while changenumbers are the same unless something in the pad has been changed. Copyand change numbers always increase, so the newer copy can always bedetermined. Finally, the SmartPad calculator also includes a clock, sothat the date and time of the last change can be recorded in each pad.

To access a saved pad, the user inserts the removable memory into theslot. The SmartPad calculator displays a table of contents (filelisting) for the memory. The listing shows the title, copy number, anddate or change number of each saved pad. In addition, on top is a titlearea for identifying the removable memory and at the right is a commandmenu. To select a saved pad, the user taps its listing.

To save the current pad onto removable memory, the user taps the "save"command. The SmartPad calculator warns the user of anything unusual(e.g., the pad has already been saved). If there is no room, theSmartPad calculator asks if older copies can be deleted first. To load apad from removable memory (making it the current one), the user selectsthe pad and then taps the "load" command. If the current pad has unsavedchanges, the SmartPad calculator saves it first. The SmartPad calculatorwarns of anything unusual (for example, not loading the most recentlysaved copy). If the pad was created on a different device, the SmartPadcalculator checks if the user is the author. If so, the author'straining for the handwriting recognizer is loaded as well, so the newdevice need not be retrained.

Creation and Deletion

To create a new (empty) pad, the user inserts the removable memory andthen taps the "new" command. If necessary, the SmartPad calculator savesthe current pad first. To create a new pad on the SmartPad calculator,the user can also delete every page of the current pad and change itstitle. To delete a pad in removable memory, the user selects the pad andthen taps the "remove" command. The SmartPad calculator asks the user toverify the action, since the pad is permanently lost after deletion.

Splitting, Copying, and Merging

To split one pad into two, the user inserts the removable memory andloads the pad to split. The user then selects the pages that are goinginto the second pad. Finally, the user goes back to the pad listing andtaps the "split" command. The SmartPad calculator saves the original padwithout the selected pages, then removes the unselected pages from thecurrent pad, making it the new one. The user must change the title ofthe current pad to distinguish it from the original. The SmartPadcalculator deletes any inter-page links between the two pads. Copies ofthe numbers appear on both pads but are not linked.

Copying pages out of pad is similar to splitting. The user selects thepages to copy, goes to the pad listing, and taps the "copy" command. Theoriginal pad is saved without making any changes, and the unselectedpages are removed from the new (current) pad. To copy an entire pad, theuser need not go to the pad and select all pages. Instead, the userselects the pad in the pad listing and then taps the "copy" command.

To merge two pads into one, the user inserts the removable memory andloads the second pad. The user then goes to the pad listing, selects thefirst pad, and taps the "merge" command. The SmartPad calculator readsin the selected pad, adding its pages to the current pad. The SmartPadcalculator then goes back to the table of contents for the current pad,so the user can place the pages in the desired order.

To transfer or copy pages from one pad to another, the user first splitsoff those pages from the source pad, then merges them with thedestination pad.

Porting

File formats of the SmartPad calculator are compatible across allplatforms. Users can easily transfer pads form one device to another, aslong as both support the same kind of removable memory or datainterface. To transfer a pad by way of removable memory, the userinserts the memory in the source device and saves the pad. The memory isthen removed and inserted in the destination device, where the pad isloaded. To transfer a pad through the data interface, the two devicesare connected (with infrared transceivers, the devices are pointed ateach other). Then the "receive" command is selected on the destinationdevice, followed by the "send" command on the source device. After thetransfer is complete, the devices are disconnected.

Security

Users with sensitive information in a pad want a way to prevent othersfrom viewing the pad. Sometimes, the author of a pad is willing to havethe pad viewed by others, but not changed. To secure access to a pad,the user goes to the table of contents and taps the "secure" command.The SmartPad calculator displays the "security" page, where the userchooses the level of security and the access method. The four levels ofsecurity are: total, view, input, and none. Total security means the padcannot be seen without following the access protocol. View securitymeans the pad can be viewed, but cannot be changed without properaccess. Input security means the pad can be viewed and the input numberscan be changed, but changes to the layout or the calculations requireproper access. In all cases, changes to security require proper access.There may also be restrictions on deleting or duplicating the pads.

The SmartPad calculator might offer one or more of three access methods:passnumber, password, or signature. The passnumber method requires theuser to write in the author's "secret" number to gain access. Thepassword method presents the user with a grid of letters, wherein theuser must tap the letters in the right order to spell out the author'ssecret password. Finally, the signature method requires the user towrite the author's signature, wherein the SmartPad calculator useshandwriting recognition to verify the signature. To make forgerydifficult, the signature must be written quickly. To make sure theSmartPad calculator can recognize a valid signature, the author at somepoint must train the SmartPad calculator by repeatedly signing until thesignature is recognized most of the time.

16. PRINTING

Printed copies of the SmartPad calculator calculations provide apermanent record for backup or archiving, and allow calculations to becommunicated to others. Unless the screen 12 can be photocopied,printing requires either a built-in printer or an attached printercompatible with the data interface. Another option is to transfer thepad (by way of removable memory) to another SmartPad calculator equippedwith a printer. A SmartPad calculator on a general-purpose computerprovides the most printing options.

Printers with fully addressable pixels offer the best printouts, sincethe text of the calculations can be placed anywhere, and the graphiclabels can be printed as well. With these printers, the hard copy is areplica of the page as displayed on the screen 12. If the real paper issmaller than the screen 12 (or page), the SmartPad calculator reducesthe size of the image to fit the paper.

The SmartPad calculator might also support printing on text-onlyprinters. The SmartPad calculator makes the printout resemble the screen12 as much as practical, given the limits of the printer. This includesplacing numbers and symbols to maintain position relationships, anddrawing lines as a series of characters.

To print a page of calculations, the user goes to the table of contentsand selects the page to print, then taps the "print" command. If aprinter is built into the platform, the SmartPad calculator sends theimage to that printer. Otherwise, the image is sent out the datainterface.

17. EXPORTING

Very complex calculations may exceed the SmartPad calculator'scapabilities or degrade its performance. Users can "export" thesecalculations to a more capable application, such as a spreadsheet or aprogramming language.

To Spreadsheet

A spreadsheet is a rectangular grid of cells, wherein each cell iseither blank or contains a label, a number, or a formula and the numericresult of evaluating that formula. The main differences between theSmartPad calculator and a spreadsheet are: (1) spreadsheet cells lie ongrid but the SmartPad calculator numbers can be placed anywhere; (2)formulas in spreadsheets occupy the same cell as the numeric result, butthe SmartPad calculator formulas are next to the results; (3) andspreadsheet labels are text, but the SmartPad calculator labels aregraphics.

To export a calculation to a spreadsheet, the user first goes to thepad's table of contents and selects the page to be exported. The userthen taps the "spreadsheet" command. The SmartPad calculator creates aspreadsheet file equivalent to the selected page. If the user selectsanother page and exports it, the two pages are merged into the samespreadsheet file. This allows inter-page link to be resolved within thespreadsheet. When a SmartPad calculator calculation is converted to aspreadsheet, every number in the calculation is placed in one of thecells of the spreadsheet. The SmartPad calculator picks the cells tomaintain the position relationships between the numbers. After that,formulas are generated for every result cell. Finally, any otherrequired information (such as number cell formats) is generated and thefile is output. The file is output in an industry-standard format.

If the SmartPad calculator platform has a keyboard, the user can replacethe ink labels with real text. The SmartPad calculator could then exportthe labels to the spreadsheet. As with numbers, the SmartPad calculatorpicks cells to hold the labels to maintain position relationships.

Programming

A program is an algorithm expressed in a programming language, whereinthe algorithm specifies changes in the values that are stored invariables. The main differences between the SmartPad calculator and aprogramming language are: (1) numbers in a program must be stored innamed variables, but the SmartPad calculator numbers are stored atunnamed positions on the display; and (2) a program has no labels otherthan the variable identifiers, which lack full expressiveness.

To export a calculation to a program, the user does the same thing asexporting to a spreadsheet, except the "program" command is tappedinstead of the "spreadsheet" command. In addition, the user is expectedto provide variable names for every number, which makes a keyboardhighly desirable.

When a SmartPad calculator calculation is converted to a program, avariable is created for each number, wherein the names of the variablesare provided by the user. After that, assignment statements for eachvariable are generated in the appropriate order. Finally, the file isoutput as plain text, suitable for input to any compiler. The program isgenerated in the C programming languages; other languages may also besupported.

18. PLOTTING

The SmartPad calculator might support plotting. Plots can either becontinuous or empirical. A continuous plot is a curve showing the resultof a calculation as a function of an input, wherein the user identifieswhich number is the input and which is the result. An empirical plot isa series of bars or dots showing the values from a list or table,wherein the user enters the values in a table, or generates by acalculation.

Using the SmartPad calculator, the user draws the axes of the plot,e.g., by drawing a big "L" character. In response, the SmartPadcalculator re-draws the axes and formalizes them by displaying arectangle graphic. To indicate the plotting range, the user writesnumbers at the axes ends and the SmartPad calculator fills inintermediate values on the axes. Thereafter, the SmartPad calculatordraws the plot by adding lines or dots to the screen 12.

For continuous plots, the user first identifies variables for the plotby drawing copy operation arrows. The SmartPad calculator then performsthe plot generation, wherein it evaluates each point. Finally, theSmartPad calculator displays the plot by drawing lines through thepoints of the plot.

For empirical plots, the user first tabulates the desired plot values bycreating a list of numbers on the screen 12 of the SmartPad calculator.The SmartPad calculator then displays the plot by drawing lines, dots orbars through the points of the plot.

For plotting operations, the SmartPad calculator also supports such"cleanup" functions as re-sizing the plot, changing the range of theplot, changing the calculations, and changing the data displayed on theplot.

19. TEACHING

A teaching version of the SmartPad calculator may provide instructionand practice in manual arithmetic to primary students. The screen 12displays problems in elementary arithmetic to the student. The studentsolves the problem manually by writing on the screen 12, as students donow on paper. Every step of the manual process is recognized by theteaching SmartPad calculator. The teaching SmartPad calculator noticeswhen a student is doing something wrong and provides immediate feedback.

A teacher interface portion for the SmartPad calculator allows theteacher to set problem levels, generate problems, and review thestudent's work on the problems. A student interface for the SmartPadcalculator allows the student to display the problems, display on-linetutorials or instruction, respond to the problems, etc.

20. CHECKBOOK

A checkbook version of the SmartPad calculator simulates a manualcheckbook register, using handwriting recognition to automate manyfunctions. The checkbook version of the SmartPad calculator holds thechecks, and automatically turns on when the checkbook is opened. Checksare inscribed on the screen 12 of the SmartPad calculator using a realpen instead of a stylus. The screen 12 of the SmartPad calculator actsas a "carbon copy" for the information written on the check, wherein theSmartPad calculator senses the position of the pen on the checks. Theamount and date on the check are recognized by the SmartPad calculatorand are entered into a check register therein. The name of the recipientis not recognized, but is entered into the check register as written.The SmartPad calculator automatically updates the account balance,displays the current date, shows a calendar, does simple calculations,reminds the user when regular payments are due, permits manualcorrections, and issue warnings if something looks wrong (e.g.,incorrect date). Finally, the SmartPad calculator allows the checkregister to be printed.

21. DATA STRUCTURES

The processing circuit requires inputs, generates outputs, requiresconstants, and uses variables to control the operation of the SmartPadcalculator. Some of this data is described in more detail below. Thoseskilled in the art will recognize that other forms of data could be usedas well.

Inputs

current time

screen turned on or off

pen touching screen or not

pen position

Outputs

display image (bitmap of screen)

ink image (bitmap of written ink; overlays display image)

Constants

screen width and height

page area height (at bottom of screen)

title/palette width and height (at top left corner of screen)

number of palette rows and columns

palette column width and row height

for each palette row

for each palette column

palette symbol

symbol image (bitmap of symbol displayed on screen)

menu width and height (at top right corner of screen)

number of menu rows and columns

menu column width and row height

for each menu row

for each menu column

menu item image (bitmap of menu item displayed on screen)

blink duration (for blinking selected menu items)

flash on and off duration (for flashing selected objects)

animate on and off duration (for animating calculations)

character form factor (width/height ratio)

counter form factor (width/height ratio)

result line form factor (height)

arrowhead form factor (width and height)

scroller form factor (inner and outer size)

selection handle size

caret width and height

for each symbol:

represents action (something to do) or object (something to create)

default model (for handwriting recognition)

Variables

displaying title or palette

scroll position (point displayed at top left corner of page area)

initial pen position (where pen first touched screen, last time it did)

ink written (list of points defining path of pen, last time pen touchedscreen)

symbol written: none, character A-Z, digit 0 to 9, period, plus, minus,times, divide, left parenthesis, right parenthesis, line, arrow,compass, frame, scratch-out, pigtail, caret (horizontal, vertical), wave(horizontal, vertical)

for each symbol:

model (for handwriting recognition; this includes training to date)

list of objects on page:

kind of object: label, scroller, character, result line, arrow, number,counter, and calculation (horizontal, vertical, copy)

original ink (list of points; copy of written ink creating this object;not for numbers or calculations; may be purged when out of memory)

selected or not (not for calculations)

area (rectangular region this object occupies; not for arrows)

graphic form (list of points; stylized representation of this object;not for characters, numbers, or calculations)

moved or not (if moved, nearby objects need to be aligned at nextavailable opportunity; not for calculations)

drawn or not (if not drawn, object needs to be drawn at next availableopportunity; not for calculations)

children (list of component objects; only for numbers, counters, andcalculations; numbers and counters only contain characters; calculationscontain characters, numbers, counters, result lines, and arrows)

parent (number, counter, or calculation that includes this object as achild; not for labels, scrollers, or calculations)

calculator (calculation that sets this object's value; only for numbers)

list of copiers (copy calculations that use this object's value; onlyfor numbers and counters)

value (only for characters, numbers, and counters)

computed or not (if not computed, value needs to be computed at nextavailable opportunity; only for numbers)

list of labels in title:

graphic form (list of points)

animated calculation (which calculation is currently being animated)

animated object (which object in animated calculation is being flashed)

animate on or off (animated object visible or not)

animate time (time of next change to visibility)

selection area (rectangular region enclosing all selected objects onpage)

flash on or off (selected objects visible or not)

flash time (time of next change to visibility)

handles displayed or not

undo page list (copy of list of objects on page, to allow undo)

undo title list (copy of list of labels in title)

undo selection area (copy of selection area)

22. FLOW CHARTS

FIGS. 40-58 are flow charts describing the logic performed by theprocessing circuit in the SmartPad calculator. Those skilled in the artwill recognize that this logic may be implemented in either hardware orsoftware. Those skilled in the art will also recognize that the logiccould be altered without departing from the scope of the presentinvention.

FIG. 40 is a flow chart describing the logic of the Program function inthe preferred embodiment of the SmartPad calculator. The Programfunction provides the main processing loop for the SmartPad calculator.Block 4002 represents the start of the Program function. Block 4004performs the Initialize function, as described in more detail in FIG.41. Block 4006 performs the Update-Page function, described in moredetail in FIG. 42. Block 4008 is a decision block that determines if thescreen is turned on. If the screen is not turned on in Block 4008,control is transferred back to Block 4006. If the screen is turned on inBlock 4008, then control is transferred to Block 4010. Block 4010 is adecision block that determines whether the pen or stylus is touching thescreen. If the pen is not touching the screen, then control istransferred to Block 4006. If the pen is touching the screen, thencontrol is transferred to Block 4012. Block 4012 performs theRespond-to-Pen function, as described in more detail in FIG. 43.

FIG. 41 is a flow chart describing the logic of the Initialize functionin the SmartPad calculator. Block 4102 represents the start of theInitialize function. Block 4104 clears the display image structure.Block 4106 clears the ink image structure. Block 4108 draws the title onthe screen. Block 4110 draws the menu on the screen. Block 4112 sets astatus flag which indicates that the title has been displayed. Block4114 resets the scroll position of the screen, i.e., the point displayedat the top left corner of the page area on the screen. Block 4116 clearsthe ink written structure. Blocks 4118-4124 loop to set all Symbols to aDefault Model. Block 4126 sets the list of objects on the page to"none". Block 4128 sets the list of labels in the title to "none". Block4130 sets animate calculation to "none". Block 4132 sets the SelectionArea to "none". Block 4134 sets the undo page list, undo title list, andundo selection area to "none". Block 4136 represents the termination ofthe Initialize function.

FIG. 42 is a flow chart describing the logic of the Update-Page functionin the SmartPad calculator. Block 4202 represents the start of theUpdate-Page function. Block 4204 finds the first object on the page thathas moved. Block 4206 is a decision block that determines whether anyobject has moved on the page. If an object has moved, then control istransferred to Block 4208. Block 4208 aligns any related objects andcontrol is returned at Block 4210. If Decision Block 4206 indicates thatan object has not moved, then Block 4212 finds the first object on thepage that is not drawn. Block 4214 is a decision block that determineswhether there is any object not drawn. If an object is not drawn, thencontrol is transferred to Block 4216. Block 4216 displays the object onthe screen. Control is then transferred to Block 4210, which terminatesthe Update-Page function. If Decision Block 4214 indicates that all theobjects have been drawn, then control is transferred to Block 4218.Block 4218 is a decision block that determines whether there is aselection area. If there is a selection area, then control istransferred to Block 4220. Block 4220 is a decision block thatdetermines if handles are displayed for the selection. If handles arenot displayed, then control is transferred to Block 4222. Block 4222inverts the selection handles on the screen, and control is transferredto Block 4210. Block 4210 terminates the Update-Page function. If Block4218 indicates that there is no selection area, or if Block 4220indicates that handles are displayed, then control is transferred toBlock 4224. Block 4224 is a decision block that determines whether thereis an animated calculation. If there is an animated calculation, thencontrol is transferred to Block 4226. Block 4226 is a decision blockthat determines whether the current time is later than the animate time.If so, then control is transferred to Block 4228. Block 4228 animatesthe calculation, and then control is transferred to Block 4230. Block4230 terminates the Update-Page function. If Decision Block 4226indicates that the current time is not later than the animate time, thencontrol is transferred to Block 4230. Block 4230 then terminates theUpdate-Page function. If Decision Block 4224 indicates that there is noanimated calculation, then control is transferred to Block 4232. Block4232 finds a number ready to calculate, by searching every object on thepage that is a number, where the number has not been computed and theobject has a calculator symbol. Block 4234 is a decision block thatdetermines whether there is a number ready to calculate. If so, thencontrol is transferred to Block 4236 which performs the Calculatefunction, as described in more detail in FIGS. 58A and 58B. After theCalculate function is performed, control is transferred to Block 4230and the Update-Page function is terminated. If Decision Block 4234indicates that there is no number ready for calculation, then control istransferred to Block 4238. Block 4238 is a decision block thatdetermines whether there is a selection area. If so, then control istransferred to Block 4240. Block 4240 is a decision block thatdetermines whether the current time is later than the flash time. IfDecision Block 4240 indicates that the current time is later than theflash time, then control is transferred to Block 4242. Block 4242flashes the selection area on the screen. Control is then transferred toBlock 4230 and the Update-Page function is terminated. If Decision Block4238 indicates that there is no selection area, then control istransferred to Block 4230 and the Update-Page function is terminated.Similarly, if Decision Block 4240 indicates the current time is notlater than the flash time, then control is transferred to Block 4230 andthe Update-Page function terminates.

FIG. 43 is a flow chart describing the logic of the Respond-to-Penfunction in the SmartPad calculator. Block 4302 represents the start ofthe Respond-to-Pen function. Block 4304 reads the initial pen positionon the screen. Block 4306 is a decision block that determines whetherthe pen is in the title or the palette area of the screen. If so, thecontrol is transferred to Block 4308. Block 4308 is a decision blockthat determines whether the title is displayed. If so, then control istransferred to Block 4310. Block 4310 performs the Handle-Pen-In-Titlefunction, as described further in FIG. 44. Control is then transferredto Block 4312 which erases the ink image. Block 4314 represents thetermination of the Respond-to-Pen function. If Decision Block 4308determines that the title is not displayed, then control is transferredto Block 4316. Block 4316 performs the Handle-Pen-In-Palette function,as described in more detail in FIGS. 45A and 45B. Control is thentransferred to Block 4312 which erases the ink image. Block 4314terminates the Respond-to-Pen function. If Decision Block 4306 indicatesthat the pen is not in the title or palette area of the screen, thencontrol is transferred to Block 4318. Block 4318 is a decision blockthat determines whether the pen is in the menu area of the screen. IfDecision Block 4318 determines that the pen is in the menu area of thescreen, then control is transferred to Block 4320. Block 4230 performsthe Handle-Pen-In-Menu function, as described in more detail in FIGS.46A and 46B. After this function is performed, control is transferred toBlock 4312 which erases the ink image. Block 4314 represents thetermination of the Respond-to-Pen function. If Decision Block 4318indicates the pen is not in the menu area of the screen, then control istransferred to Block 4322. Block 4322 performs the Handle-Pen-In-Pagefunction, as described in more detail in FIGS. 47A and 47B. After thisfunction is performed, control is transferred to Block 4312 which erasesthe ink image. Block 4314 represents the termination of theRespond-to-Pen function.

FIG. 44 is a flow chart describing the logic of the Handle-Pen-In-Titlefunction in the SmartPad calculator. Block 4402 represents the start ofthe Handle-Pen-In-Title function. Block 4404 saves the current data forany potential Undo functions. Block 4406 empties the ink writtenstructure, adds the initial pen position to the ink written structure,and, while the pen is touching the screen, reads the pen position, drawsa line in the ink image from the last point in the ink written structureto the pen position, and if the last two points in the ink writtenstructure are in line with the pen position, then the last point in theink written structure is set to the pen position so that every pointalong a straight line is not saved, otherwise the pen position is addedto the ink structure to save the pen position for curving, i.e, nonstraight, lines. Block 4408 performs the handwriting recognitionfunction to determine what symbol was written by the user on the screen.Block 4410 is a decision block that determines whether the writtensymbol is an erase or delete symbol. If Decision Block 4410 determinesthat the written symbol is an erase or delete symbol, then control istransferred to Block 4412. Block 4412 computes the area enclosing thewritten ink on the screen. Block 4414 computes an intersection of theenclosed area and the title area. Block 4416 is a decision block thatdetermines whether the intersection includes most of the title area. Ifso, then control is transferred to Block 4418. Block 4418 empties thelist of labels in the title and Block 4420 erases the title area of thedisplay image structure. Control is then transferred to Block 4222 whichterminates the Handle-Pen-In function. If Decision Block 4410 determinesthat the written symbol is not an erase or delete symbol, or if DecisionBlock 4416 determines that the intersection does not include most of thetitle area, then control is transferred to Block 4424. Block 4424 adds anew label to the list of labels in the title. Block 4426 sets the newlabel's graphic form in the list to a copy of the written ink. Block4428 draws the new label's graphic form on the screen. Block 4422terminates the Handle-Pen-In-Title function.

FIGS. 45A and 45B together are a flow chart describing the logic of theHandle-Pen-In-Palette function in the SmartPad calculator. Block 4502represents the start of the Handle-Pen-In-Palette function. Block 4504determines the row and column of the palette at the initial penposition. Block 4506 blinks the area of the palette at the determinedrow and column. Block 4508 sets the current symbol to the palette symbolat the determined row and column. Block 4510 counts the number ofobjects on the screen that have been selected by the user. Block 4512 isa decision block that determines whether any objects are selected on thescreen. If one or more objects are selected, then control is transferredto Block 4514; otherwise, control is transferred to Block 4534. Block4514 is a decision block that determines whether one or a plurality ofobjects have been selected on the screen. If one object has beenselected, then control is transferred to Block 4516. Block 4516 findsthe object on the page that has been selected by the user. Block 4518 isa decision block that determines whether the selected object has itsoriginal "ink". If so, then control is transferred to Block 4520;otherwise, control is transferred to Block 4522. Block 4520 updates thesymbol's model with the selected object's original "ink". Block 4522saves the current data in case the user later invokes the Undo function.Block 4524 erases the selected object from the screen and removes itfrom the calculation. Block 4526 is a decision block that determineswhether the symbol represents an action. If so, then control istransferred to Block 4528. Block 4528 represents the Do-Action functionfor the selected symbol, as described in more detail in FIGS. 49A and49B. Block 4530 represents the termination of the Handle-Pen-In-Palettefunction. If Decision Block 4526 determines that the symbol does notrepresent an action, then control is transferred to Block 4532. Block4532 performs the Add-Object function, as described in more detail inFIGS. 50A, 50B, 50C, and 50D. If Decision Block 4512 determines that noobjects are selected, then control is transferred to Block 4534. Block4534 is a decision block that determines whether there is written ink onthe screen. If not, then control is transferred to Block 4526;otherwise, control is transferred to Block 4536. Block 4536 updates thesymbol's model with the written ink. Block 4538 performs the Undofunction which reverses the last change made and control is thentransferred to Block 4526.

FIGS. 46A and 46B together are a flow chart describing the logic of theHandle-Pen-In-Menu function in the SmartPad calculator. Block 4602represents the start of the Handle-Pen-In-Menu function. Block 4604determines the row and column of the menu at the initial pen position.Block 4606 blinks the area of the menu at the determined row and column.Block 4608 is a decision block that determines whether the row andcolumn represents the Undo menu item. If so, then control is transferredto Block 4610. Block 4610 performs the Undo function which reverses theeffects of the last change made. Control is then transferred to Block4612 that terminates the Handle-Pen-In-Menu function. If Decision Block4608 determines that the row and column do not represent the Undo menuitem, then control is transferred to Block 4614. Box 4614 is a decisionblock that determines whether the row and column represent the Trainmenu item. If so, then control is transferred to Block 4616. Block 4616erases the title area of the display image and control is transferred toBlock 4618. Box 4618 is a decision block that determines whether or notthe palette is displayed. If so, then control is transferred to Block4620. Block 4620 draws a title on the screen and Block 4622 sets thevariable that indicates that the title is displayed. Control is thentransferred to Block 4612 and the Handle-Pen-In-Menu function isterminated. If Decision Block 4618 determines that the palette is notdisplayed, then Block 4624 draws the palette and Block 4626 sets thestatus variable indicating that the palette is displayed. Control isthen transferred to Block 4612 and the Handle-Pen-In-Menu function isterminated. If Decision Block 4614 determines that the row and columndoes not represent the Train menu item, then control is transferred toBlock 4612 which terminates the Handle-Pen-In-Menu function.

FIGS. 47A and 47B together are a flow chart describing the logic of theHandle-Pen-In-Page function in the SmartPad calculator. Block 4702represents the start of the Handle-Pen-In-Page function. Block 4704 is adecision block that determines whether the initial pen position is in aselection area. If not, then control is transferred to Block 4706 whichfinds a scroller object on the page displayed at the initial penposition. Otherwise control is transferred to Block 4708. Block 4708 isa decision block that determines whether the initial pen position isnear the top left corner of the selection area. If so, Block 4710 setsthe pen position to the initial position, and while the pen is touchingthe screen waits for a new pen position so that it can compute a newselection area from the top left of the selected object. Block 4712 is adecision block that determines whether the initial pen position is nearthe top right corner of the selection area. If so, Block 4714 sets thepen position to the initial position, and while the pen is touching thescreen waits for a new pen position so that it can compute a newselection area from the top right of the selected object. Block 4716 isa decision block that determines whether the initial pen position isnear the bottom left corner of the selection area. If so, Block 4718sets the pen position to the initial position, and while the pen istouching the screen waits for a new pen position so that it can computea new selection area from the bottom left. Block 4720 is a decisionblock that determines whether the initial pen position is near thebottom right corner of the selection area. If so, Block 4722 sets thepen position to the initial position, and while the pen is touching thescreen waits for a new pen position so that it can compute a newselection area from the bottom right of the selected object. Block 4724is a decision block that determines whether the initial pen position iswithin the selection area. If so, Block 4726 sets the pen position tothe initial position, and while the pen is touching the screen waits fora new pen position so that is can compute a new selection area. Block4728 is a decision block that determines whether there is a "scroller"object displayed on the screen. If so, Block 4730 scrolls the page.Otherwise, Block 4732 performs the Handle-Written-Symbol function, asdescribed in more detail in FIG. 48. Finally, Block 4734 represents atermination of the Handle-Pen-In-Page function.

FIG. 48 is a flow chart describing the logic of theHandle-Written-Symbol function in the SmartPad calculator. Block 4802represents the start of the Handle-Written-Symbol function. Box 4804empties the ink written structure, adds the initial pen position to theink written structure, and, while the pen is touching the screen, readsthe pen position, draws a line in the ink image from the last point inthe ink written structure to the pen position, and if the last twopoints in the ink written structure are in line with the pen position,then the last point in the ink written structure is set to the penposition so that every point along a straight line is not saved,otherwise the pen position is added to the ink structure to save the penposition for curving, i.e, non straight, lines. Block 4806 performs thehandwriting recognition function to determine the symbol written by theuser on the display. Block 4808 is a decision block that determineswhether the symbol written by the user on the screen is a lasso. If so,control is transferred to Block 4810 which resets the selection;otherwise, Block 4812 saves the data for an Undo function. Block 4814 isa decision block that determines whether the symbol written by the useron the display was recognizable. If not, then control is transferred toBlock 4816 which adds a new label to the page. If the symbol wasrecognizable, then control is transferred to Block 4818. Block 4818 is adecision block that determines whether the symbol written by the user onthe screen represents an action. If so, then Block 4820 performs theDo-Action function for the symbol, as described in more detail in FIGS.49A and 49B; otherwise, control is transferred to Block 4822. Block 4822performs the Add-Object function, as described in more detail in FIGS.50A, 50B, 50C, and 50D. Block 4824 represents the termination of theHandle-Written-Symbol function.

FIGS. 49A and 49B together are a flow chart describing the logic of theDo-Action function in the SmartPad calculator. Block 4904 is decisionblock that determines whether the symbol is a lasso. If so, Block 4906computes, for each object on the page, the amount of area if any of theobject inside the lasso, and then identifies the object as beingselected if most of the object's area is inside the lasso. Block 4908 isa decision block that determines whether the symbol is a "scratch out".If so, Block 4910 sets the scratch out area to enclose the written inkand then removes the objects in the scratch out area. Block 4912 is adecision block that determines whether the symbol is a pigtail. If so,Block 4914 removes the objects under the pigtail and closes up the blankspace. Block 4916 is a decision block that determines whether the symbolis an up or down caret. If so, Block 4918 inserts a blank space betweenthe objects where the caret was written. Block 4020 is a decision blockthat determines whether the symbol is a left or right caret. If so,Block 4922 inserts a blank line between the lines of characters wherethe caret was written. Block 4924 is a decision block that determineswhether the symbol is a horizontal wave. If so, Block 4926 swaps theobjects under the horizontal wave. Block 4928 is a decision block thatdetermines whether the symbol is a vertical wave. If so, Block 4930swaps the objects under the vertical wave. Block 4932 represents thetermination of the Do-Action function.

FIGS. 50A, 50B, 50C, and 50D together are a flow chart describing thelogic of the Add-Object function in the SmartPad calculator. Block 5002represents the start of the Add-Object function. Block 5004 computes thearea enclosing the written "ink" on the screen. Block 5006 finds theoverwritten object on the screen. Block 5008 is a decision block thatdetermines whether the overwritten object is a label. If so, Block 5010adds a new label to the page. Block 5012 is a decision block thatdetermines whether the overwritten object is a counter and the writtensymbol is not an arrow. If so, Block 5014 applies the written symbol tothe overwritten counter. Block 5016 is a decision block that determineswhether the overwritten object is an equals character, result line, orarrow. If so, Block 5018 removes the overwritten object's parent (acalculation) from the page. Block 5020 is a decision block thatdetermines whether the overwritten object is a character or scrollerobject. If so, Block 5022 removes the overwritten object from the page.Block 5024 is a decision block that determines whether the writtensymbol is a character. If so, Block 5026 adds a new character to thepage and Block 5028 sets the new object's value to the charactercorresponding to the written symbol. Block 5030 is a decision block thatdetermines whether the written symbol is an equals character. If so,Block 5032 performs the Generate-Horizontal-Calculation function withthe new object, as described in more detail in FIG. 52; otherwise,control is transferred to Block 5034. Block 5034 is a decision blockthat determines whether the written symbol is used in a number, i.e.,whether it is a digit, a period, or a comma. If so, Block 5036 performsthe Generate-Number function with the new object, as described in moredetail in FIGS. 51A and 51B. Control is then transferred to Block 5038which terminates the Add-Object function. If Decision Block 5024determines that the written symbol is not a character, then control istransferred to Block 5040. Block 5040 is a decision block thatdetermines whether the written symbol is result line. If so, Block 5042adds a new result line to the page and Block 5044 performs theGenerate-Vertical-Calculation function for the new object, as describedin more detail in FIG. 53. Control is then transferred to Block 5038which terminates the Add-Object function. If the written symbol is not aresult line, then control is transferred from Decision Block 5040 toDecision Block 5046. Block 5046 is a decision block that determineswhether the written symbol is an arrow. If so, Block 5048 adds a newarrow to the page, and Block 5050 performs the Generate-Copy-Calculationfunction with the new object, as described in more detail in FIGS. 54A,54B, and 54C. Control is then transferred to Block 5038 and theAdd-Object function terminates. Block 5052 is a decision block thatdetermines whether the symbol is a square. If so, Block 5054 adds a newcounter to the page. Control is then transferred to Block 5038 whichterminates the Add-Object function. Block 5056 is a decision block thatdetermines whether the written symbol is a compass. If so, Block 5058adds a new scroller to the page and control is then transferred to Block5038 which terminates the Add-Object function.

FIGS. 51A and 51B together are a flow chart describing the logic of theGenerate-Number function in the SmartPad calculator. Block 5102represents the start of the Generate-Number function. Block 5104 setsthe search area to the character's area expanded in the left and rightdirection. Block 5106 sets the status flag to indicate that no numberhas been found. Block 5108 initializes the loop to start with the firstobject on the page. Block 5110 is a decision block that determineswhether the current object is a number. If so, Block 5112 computes theintersection of the object's area and the search area. Block 5114 is adecision block that determines whether there is an intersection betweenthe object's area and the search area. If so, control is transferred toBlock 5116. Block 5116 is a decision block that determines whether theobject is closer to the left side of the character's area than thenumber. If so, Block 5118 sets the number to the object. Block 5120 is adecision block that controls the end of loop by determining whether ornot the last object has been examined. If not, Block 5122 sets the nextobject on the page for examination and control is transferred to Block5110. After the last object has been examined, as determined by DecisionBlock 5120, then control is transferred to Block 5124. Block 5124 is adecision block that determines whether a number has been found in theloop. If not, Block 5126 adds a new number to the page. Block 5128 addsthe character to the number. Block 5130 terminates the Generate-Numberfunction.

FIG. 52 is a flow chart describing the logic of theGenerate-Horizontal-Calculation function in the SmartPad calculator.Block 5202 represents the start of the Generate-Horizontal-Calculationfunction. Block 5204 adds a new horizontal calculation to the page.Block 5206 adds the character to the new calculation. Block 5208performs the Find-Object-To-Add-To-Horizontal-Calculation function withthe new calculation, as described in more detail in FIGS. 55A and 55B.Block 5210 is a decision block that determines whether there is anobject to add. If so, Block 5212 adds the object to the horizontalcalculation. Block 5214 performs the Create-Calculation-Result function,as described in more detail in FIG. 57. Block 5216 terminates theGenerate-Horizon Calculation function.

FIG. 53 is a flow chart describing the logic of theGenerate-Vertical-Calculation function in the SmartPad calculator. Block5302 represents the start of the Generate-Vertical-Calculation function.Block 5304 adds a new vertical calculation to the currently displayedpage. Block 5306 adds the result line to the new calculation. Block 5308performs the Find-Object-To-Add-To-Vertical-Calculation function, asdescribed in more detail in FIGS. 56A, 56B, and 56C. Block 5310 is adecision block that determines whether there is another object to add tothe vertical calculation. If so, Block 5312 adds the object to thevertical calculation. If not, Block 5314 performs theCreate-Calculation-Result function for the new calculation, as describedin more detail in FIG. 57. Block 5316 terminates theGenerate-Vertical-Calculation function.

FIGS. 54A, 54B, and 54C together are a flow chart describing the logicof the Generate-Copy-Calculation function in the SmartPad calculator.Block 5402 represents the start of the Generate-Copy-Calculationfunction. Block 5404 sets the source number to "none". Block 5406initializes the processing loop, which begins with the first object onthe page. Block 5408 is a decision block that determines whether theobject is a number or a counter. If so, Block 5410 computes the centerof the object's area. Block 5412 is a decision block that determineswhether the center is close enough to the first point of written ink. Ifso, control is transferred to Block 5414. Block 5414 is a decision blockthat determines whether the center of the object's area is closer thanthe source number's center. If so, Block 5416 sets the source number tothis object and control is transferred to Block 5418. If the center ofthe object's area is not close enough to the first point of written ink,or if the center of the object's area is not closer than the sourcenumber's center, then control is transferred to Block 5418. Block 5418is a decision block that determines whether the loop has processed allobjects on the page. If not, the next object is selected by Block 5420and control is transferred to decision Block 5408 for additionalprocessing. If the last object has been processed, then control istransferred to Block 5422. Block 5422 is a decision block thatdetermines whether there is a source number. If yes, Block 5424 adds anew copy calculation to the page. Block 5426 adds the source number tothe calculation. Block 5428 adds the calculation to the source number'slist of copiers. Block 5430 adds the arrow to the calculation. Block5432 performs the Create-Calculation-Result function for the calculationat the last point of written ink, as described in more detail in FIG.57. Block 5434 routes the arrow from the source number's area to theresult's area. Block 5436 terminates the Generate-Copy-Calculationsfunction. If Decision Block 5422 determines that there is not a sourcenumber, then Block 5438 removes the arrow from the page and Block 5436terminates the Generate-Copy-Calculation function.

FIGS. 55A and 55B together are a flow chart describing the logic of theFind-Object-To-Add-To-Horizontal-Calculation function in the SmartPadcalculator. Block 5502 represents the start of theFind-Object-To-Add-To-Horizontal-Calculation function. Block 5504 setsthe search area to the area to the left of the calculation. Block 5506sets the object to "none". Block 5508 initializes the processing loop byselecting the first object on the page. Block 5510 is a decision blockthat determines whether the current object is a character, number, orcounter. If so, control is transferred to Block 5512. Block 5512 is adecision block that determines whether the object has a parent. If not,Block 5514 computes an intersection between the object's area and thesearch area. Block 5516 is a decision block that determines whetherthere is an intersection between the object's area and the search area.If so, then control is transferred to Block 5518. Block 5518 is adecision block that determines whether there is an object to add to thecalculation. If so, then control is transferred to Block 5520. Block5520 is a decision block that determines whether the current object isto the right of the object to add. If so, or if Decision Block 5518determines that there is no object to add, Block 5522 identifies theobject as the one to add. Block 5524 is a decision block that terminatesthe loop. If there are more objects on the page, then control istransferred to Block 5526 which selects the next object to process.Otherwise, control is transferred to Block 5528 which terminates theFind-Object-To-Add-To-Horizontal-Calculation function.

FIGS. 56A, 56B, and 56C together are a flow chart describing the logicof the Find-Object-To-Add-To-Vertical-Calculation function in theSmartPad calculator. Block 5602 represents the start of theFind-Object-To-Add-To-Vertical-Calculation function. Block 5604 sets theprimary search area to the area to the left of the first child of thecalculation. Block 5606 sets the secondary search area to the area abovethe calculation. Block 5608 sets the object to add to "none". Block 5610initiates the processing loop to start with the first object on thepage. Block 5612 is a decision block that determines whether the currentobject is a character, number or counter. If so, then control istransferred to Block 5614. Block 5614 is a decision block thatdetermines whether the current object has a parent. If not, Block 5616computes the intersection of the object's area and the primary searcharea. Block 5618 is a decision block that determines whether there is anintersection between the object's area and the primary search area. Ifnot, Block 5620 computes the intersection of the object's area and thesecondary search area. Block 5622 is a decision block that determineswhether there is an intersection between the object's area and thesecondary search area. If there is an intersection between the object'sarea and either the primary search area or the secondary search area,then control is transferred to Block 5624. Block 5624 is a decisionblock that determines whether there is an object to add to thecalculation. If so, then control is transferred to Block 5626. Block5626 is a decision block that determines whether the current object isbelow the object to add. If not, then control is transferred to Block5628. Block 5628 is a decision block that determines whether the currentobject is to the right of the object to add and not above it. If theobject is to the right of the object to add and not above it, or if theobject is below the object to add, or if there is no object to add, thencontrol is transferred to Block 5630 which identifies the current objectas the object to add. Control is then transferred to Block 5632. Block5632 is a decision block that determines whether every object on thepage has been processed. If so, Block 5634 terminates theFind-Object-To-Add-To-Vertical-Calculation function. If not, control istransferred to Block 5636 which selects the next object on the page forprocessing.

FIG. 57 is a flow chart describing the logic of theCreate-Calculation-Result function in the SmartPad calculator. Block5702 represents the start of the Create-Calculation-Result function.Block 5704 adds a new character to the page at the desired area. Block5706 sets the new character's value to blank. Block 5708 adds a newnumber (the calculated result) to the page at the desired area. Block5710 adds the new character to the new number. Block 5712 adds the newnumber to the calculation. Block 5714 sets the new number's calculatorto the calculation. Block 5716 represents the termination ofCreate-Calculation-Result function.

FIGS. 58A and 58B together are a flow chart describing the logic of theCalculate function in the SmartPad calculator. Block 5802 represents thestart of the Calculate function. Block 5804 is a decision block thatdetermines whether the number has a calculator. If so, then control istransferred to Block 5806. Block 5806 is a decision block thatdetermines whether the number's calculator is horizontal. If so, thenBlock 5808 computes the result by evaluating the horizontal formula. Ifnot, then control is transferred to Block 5810. Block 5810 is a decisionblock that determines whether the number's calculator is vertical. Ifso, then Block 5812 computes the result by evaluating the verticalformula. If the number's calculator is neither vertical nor horizontal,then Block 5814 computes the result using the value of the first childof the number's calculator. After the result has been computed, Block5816 forces the number's value to the result. Block 5818 sets theanimated calculation variable to identify the number's calculator. Block5820 sets the animated object variable to identify the first child ofthe calculation. Block 5822 turns the animation flag on. Block 5824 setsthe animate time to the current time. If the number does not have acalculator, or after the animate time has been set to the current time,Block 5826 sets the text to the sequence of values of the number'schildren, i.e., the characters. Block 5828 sets the number's value to adecoding of the text as a number. Block 5830 turns the number's computedflag on. Block 5832 represents the termination of the Calculatefunction.

23. CONCLUSION

This concludes the description of the preferred embodiment of theinvention. In summary, the present invention discloses a pen-basedcalculator which recognizes handwritten input. The calculator consistsof a display simulating a sheet of paper, and a stylus simulating a pen.The user writes a calculation on the calculator as if it were a piece ofscratch paper. The calculator uses handwriting recognition to identifythe various elements of the calculation, performs the calculation, andthen displays the result at an appropriate location.

The foregoing description of the preferred embodiment of the inventionhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of theinvention be limited not by this detailed description, but rather by theclaims appended hereto.

What is claimed is:
 1. A method of performing calculations in a calculator having an electronic input surface, an electronic monitor, and a processing circuit coupled to the electronic input surface and the electronic monitor, the method comprising the steps of:(a) recording movements of a pointing element in the processing circuit, as the pointing element is traced across the electronic input surface; (b) recognizing the recorded movements of the pointing element as characters in the processing circuit; (c) converting the characters into a first mathematical expression comprised of operands and operators in the processing circuit; (d) displaying the first mathematical expression on the electronic monitor; (e) performing calculations indicated by the displayed first mathematical expressions in the processing circuit; (f) displaying a result of the performed calculations on the electronic monitor; and (g) logically linking the first mathematical expression to a second mathematical expression inscribed on the electronic input surface.
 2. The invention as set forth in claim 1, wherein the operands comprise symbols.
 3. The invention as set forth in claim 1, wherein the operands comprise digits.
 4. The invention as set forth in claim 3 above, further comprising the step of recognizing numbers from the relative placement of the digits, so that when the digits are traced horizontally in close proximity to one another on the electronic input surface, they are considered to be a single number.
 5. The invention as set forth in claim 1 above, further comprising the step of recognizing mathematical expressions traced horizontally and vertically on the electronic input surface.
 6. The invention as set forth in claim 1 above, further comprising the step of computing a result for the calculations when the user traces a result operator on the electronic input surface.
 7. The invention as set forth in claim 6 above, wherein the result operator is an equal sign in a horizontal mathematical expression.
 8. The invention as set forth in claim 6 above, wherein the result operator is a result line in a vertical mathematical expression.
 9. The invention as set forth in claim 1 above, further comprising the step of animating expressions on the electronic input surface as they are being calculated.
 10. The invention as set forth in claim 1 above, further comprising the step of accepting corrections in the mathematical expressions traced by the pointing element in the electronic input surface.
 11. The invention as set forth in claim 1 above, further comprising the step of logically linking together a plurality of mathematical expressions inscribed on the electronic input surface.
 12. The invention as set forth in claim 1 above, wherein the first and second mathematical expressions are linked in response to their proximity to one another on the electronic input surface.
 13. The invention as set forth in claim 1 above, wherein the first and second mathematical expressions are linked in response to a user tracing a linking operator on the electronic input surface.
 14. The invention as set forth in claim 13 above, wherein the linking operator is an arrow having a tail proximal the first mathematical expression and a head proximal the second mathematical expression.
 15. The invention as set forth in claim 14 above, wherein a result from the first mathematical expression is an opened in the second mathematical expression.
 16. The invention as set forth in claim 14 above, wherein the logically linking stem further comprises the step of re-computing the second mathematical expression when the first mathematical expression is altered on the electronic input surface.
 17. The invention as set forth in claim 1 above, wherein the logically linking step further comprises the step of re-computing the first and second mathematical expressions logically linked together, thereby incorporating a result of the first mathematical expression into the second mathematical expression.
 18. The invention as set forth in claim 1 above, wherein the logically linking step further comprises the step of re-computing the first and second mathematical expressions logically linked together, wherein the first and second mathematical expressions are on separate pages display on the electronic monitor, thereby incorporating a result of the first mathematical expression into the second mathematical expression.
 19. The invention as set forth in claim 1 above, wherein the logically linking step further comprises the step of re-computing the first and second mathematical expressions logically connected together, wherein the first and second mathematical expressions are in separate applications executed by the processing circuit, thereby incorporating a result of the first mathematical expression into the second mathematical expression.
 20. The invention as set forth in claim 1 above, further comprising the step of accepting marks traced by the pointing element on the electronic input surface to annotate and label the recorded movements.
 21. The invention as set forth in claim 1 above, further comprising the step of accepting insertions in the mathematical expressions traced by the pointing element on the electronic input surface.
 22. The invention as set forth in claim 1 above, further comprising the step of accepting deletions in the mathematical expressions traced by the pointing element on the electronic input surface.
 23. The invention as set forth in claim 1 above, further comprising the step of accepting erasures in the mathematical expressions traced by the pointing element on the electronic input surface.
 24. A calculator having an electronic input surface, an electronic monitor, and a processing circuit coupled to the electronic input surface and the electronic monitor, comprising:(a) means for recording movements of a pointing element in the processing circuit, as the pointing element is traced across the electronic input surface; (b) means for recognizing the recorded movements of the pointing element as characters in the processing circuit; (c) means for converting the characters into a first mathematical expression comprised of operands and operators in the processing circuit; (d) means for displaying the first mathematical expression on the electronic monitor; (e) means for performing calculations indicated by the displayed first mathematical expression in the processing circuit; (f) means for displaying a result of the performed calculations on the electronic monitor; and (g) means for logically linking the first mathematical expression to a second mathematical expression inscribed on the electronic input surface.
 25. The invention as set forth in claim 24, wherein the electronic monitor is the electronic input surface.
 26. The invention as set forth in claim 24, wherein the operands comprise symbols.
 27. The invention as set forth in claim 24, wherein the operands comprise digits.
 28. The invention as set forth in claim 27 above, further comprising means for recognizing numbers from the relative placement of the digits, so that when the digits are traced horizontally in close proximity to one another on the electronic input surface, they are considered to be a single number.
 29. The invention as set forth in claim 24 above, further comprising means for recognizing mathematical expressions traced horizontally and vertically on the electronic input surface.
 30. The invention as set forth in claim 24 above, further comprising means for computing a result for the calculations when the user traces a result operator on the electronic input surface.
 31. The invention as set forth in claim 30 above, wherein the result operator is an equal sign in a horizontal mathematical expression.
 32. The invention as set forth in claim 30 above, wherein the result operator is a result line in a vertical mathematical expression.
 33. The invention as set forth in claim 24 above, further comprising means for animating expressions on the electronic input surface as they are being calculated.
 34. The invention as set forth in claim 24 above, further comprising means for accepting corrections in the mathematical expressions traced by the pointing element in the electronic input surface.
 35. The invention as set forth in claim 24 above, wherein the first and second mathematical expressions are linked in response to their proximity to one another on the electronic input surface.
 36. The invention as set forth in claim 24 above, wherein the first and second mathematical expressions are linked in response to a user tracing a linking operator on the electronic input surface.
 37. The invention as set forth in claim 36 above, wherein the linking operator is an arrow having a tail proximal the first mathematical expression and a head proximal the second mathematical expression.
 38. The invention as set forth in claim 37 above, wherein a result from the first mathematical expression is an operand in the second mathematical expression.
 39. The invention as set forth in claim 37 above, wherein the means for logically linking further comprises means for re-computing the second mathematical expression when the first mathematical expression is altered on the electronic input surface.
 40. The invention as set forth in claim 24 above, wherein the means for logically linking further comprises means for re-computing the first and second mathematical expressions logically linked together, thereby incorporating a result of the first mathematical expression into the second mathematical expression.
 41. The invention as set forth in claim 24 above, wherein the means for logically linking further comprises means for re-computing the first and second mathematical expressions logically linked together, wherein the first and second mathematical expressions are on separate pages displayed on an electronic monitor, thereby incorporating a result of the first mathematical expression into the second mathematical expression.
 42. The invention as set forth in claim 24 above, wherein the means for logically linking further comprises means for re-computing the first and second mathematical expressions logically connected together, wherein the first and second mathematical expressions are in separate applications executed by the processing circuit, thereby incorporating a result of the first mathematical expression into the second mathematical expression.
 43. The invention as set forth in claim 24 above, further comprising means for accepting marks traced by the pointing element on the electronic input surface to annotate and label the recorded movements.
 44. The invention as set forth in claim 24 above, further comprising means for accepting insertions in the mathematical expressions traced by the pointing element on the electronic input surface.
 45. The invention as set forth in claim 24 above, further comprising means for accepting deletions in the mathematical expressions traced by the pointing element on the electronic input surface.
 46. The invention as set forth in claim 24 above, further comprising means for accepting erasures in the mathematical expressions traced by the pointing element on the electronic input surface. 